Skip to content

Close performance and conformance gaps with Shen/JIT - #101

Merged
pyrex41 merged 65 commits into
mainfrom
team/all
Nov 7, 2026
Merged

pyrex41 merged 65 commits into
mainfrom
team/all

Conversation

@pyrex41

@pyrex41 pyrex41 commented Nov 6, 2026 •

Copy link
Copy Markdown
Owner

Combines five measured improvement branches (conform, boot, lists, calls, prims) on kernel 43.2, plus the fixes that integration and two review rounds needed. Head: 5a0228d. Every number below names the SHA it was measured on.

Head 3c0dc96: 5a0228d plus a clean merge of main 0c2e06f (#102, REPL EOF spec); specs on the merged head: 2399 pass, 0 fail across 38 specs.

Performance against 912596b and Shen/JIT

Instructions retired, billions (G), Apple Silicon under load (15-40), median (range) of n interleaved runs, private kernel and fasl caches. Warm means a fasl hit; first means an empty fasl dir with a warm kernel cache, which is the fair comparison with Shen/JIT on compile-heavy rows, since Shen/JIT has no cache. Measured on 67a79f3; the two commits after it (the nil check and the changelog) measured within noise of it, see below. Ratios are head/Shen-JIT.

row Shen/JIT 912596b warm 912596b first head warm head first ratio warm ratio first
startup 0.05 (0.04-0.05) n=3 0.76 (0.75-0.82) n=3 0.79 (0.78-0.83) n=3 0.45 (0.42-0.46) n=3 0.47 (0.42-0.47) n=3 9.73 10.20
fib 1.44 (1.44-1.44) n=3 5.03 (4.93-5.04) n=3 5.16 (5.12-5.17) n=3 3.36 (3.30-3.37) n=3 3.33 (3.32-3.40) n=3 2.34 2.31
loop 3.66 (3.66-3.66) n=3 2.16 (2.14-2.17) n=3 2.29 (2.26-2.30) n=3 1.86 (1.82-1.89) n=3 1.89 (1.86-1.89) n=3 0.51 0.52
nrev 18.70 (18.69-18.72) n=3 288.15 (182.32-288.81) n=3 182.45 (182.30-183.63) n=3 173.76 (173.59-173.99) n=3 173.40 (173.25-173.54) n=3 9.29 9.27
hof 1.99 (1.98-2.01) n=3 48.83 (48.09-48.87) n=3 48.10 (47.89-48.57) n=3 9.02 (8.97-9.02) n=3 9.04 (9.01-9.08) n=3 4.53 4.54
vec 3.10 (3.10-3.11) n=3 13.02 (13.02-13.03) n=3 13.26 (13.24-13.44) n=3 1.87 (1.85-1.87) n=3 1.97 (1.96-1.98) n=3 0.60 0.63
str 1.70 (1.70-1.71) n=3 4.17 (3.66-4.18) n=3 4.14 (3.91-4.14) n=3 1.06 (1.06-1.14) n=3 1.16 (1.14-1.19) n=3 0.62 0.68
prolog 7.08 (7.08-7.17) n=3 21.17 (20.34-22.62) n=12 23.07 (22.07-23.13) n=3 14.82 (14.09-17.82) n=12 14.50 (14.21-14.78) n=3 2.09 2.05
typed 0.82 (0.81-0.82) n=3 6.18 (6.17-6.18) n=3 23.00 (22.05-23.24) n=3 1.34 (1.22-1.37) n=3 11.66 (11.44-11.66) n=3 1.64 14.27
compile 0.83 (0.83-0.83) n=3 3.30 (3.25-3.39) n=3 27.51 (27.20-28.09) n=3 0.75 (0.72-0.79) n=3 5.60 (5.55-5.77) n=3 0.90 6.73
compilecall 1.08 (1.06-1.08) n=3 3.60 (3.54-3.62) n=3 29.42 (29.40-29.72) n=3 0.79 (0.77-0.81) n=3 6.37 (6.24-6.47) n=3 0.73 5.90
kernel 3.15 (3.15-3.17) n=3 22.61 (21.88-23.50) n=3 76.51 (75.84-76.94) n=3 3.52 (3.45-3.58) n=3 49.50 (48.12-50.88) n=3 1.12 15.72
extensions 0.27 (0.27-0.27) n=3 2.07 (2.01-2.19) n=3 5.18 (5.14-5.21) n=3 1.39 (1.38-1.45) n=3 3.49 (3.44-3.58) n=3 5.16 12.97
steam 3.69 (3.67-3.72) n=3 58.94 (56.85-61.60) n=3 69.80 (69.80-69.80) n=1 48.30 (48.16-49.42) n=3 54.56 (54.56-54.56) n=1 13.09 14.79
thm13 28.22 (27.96-28.34) n=3 652.68 (652.13-653.95) n=3 944.88 (944.88-944.88) n=1 545.79 (545.50-554.35) n=3 545.83 (545.83-545.83) n=1 19.34 19.35
rcc10 20.78 (20.61-20.84) n=3 440.89 (428.48-449.38) n=3 449.80 (449.80-449.80) n=1 380.80 (374.35-393.17) n=3 385.80 (385.80-385.80) n=1 18.32 18.56
  • Ahead of Shen/JIT on instructions: loop, vec, str, and warm compile and compilecall. Level warm: kernel (1.12x).
  • The gaps that remain: first-run compile-heavy rows (6-16x), THORN (13-19x), nrev 9.3x, startup 10x, hof 4.5x.
  • Against 912596b: nrev 288 to 174 (and no slow mode), hof 48.8 to 9.0, vec 13.0 to 1.9, str 4.2 to 1.1, compile first 27.5 to 5.6, kernel warm 22.6 to 3.5, kernel first 76.5 to 49.5, THORN 12-18% lower.
  • vec is fitted to its shape: the gain needs a self-tail loop. The same work through another function or non-tail recursion is 3.1-3.6x Shen/JIT (still 1.3-2x better than 912596b). str is general: every variant beats Shen/JIT.
  • Costs: str_long +7% (5.45 to 5.81 G) from the GC stepmul 1000 default, which also takes its footprint from 78 to 47 MB; SHEN_GC=400,200 restores the old setting. The adaptive GC pause costs +67% instructions on a 3M-cell live list, for -61% footprint.
  • THORN is bimodal, and was before this PR. thm13 on 912596b: 630-667 G in 18 runs, 928-945 G in 5. On this branch: 525-582 G, or 780-847 G in about 1 run in 5. thm55 on this branch: 4087-4248 G in 7 runs, 6018 and 6026 G in 2 (9f9a79e, 67a79f3, both in gate runs); on 912596b 4957 G in 2 runs. The high/low ratio is about 1.45 on both trees. What selects the mode is unknown. Inference counts and proof hashes are exact in both modes.

Behaviour changes

  • A non-boolean test raises: the test of if and cond, both operands of and/or, and the argument of not. Before, (if 1 a b) gave a. A predicate with too few arguments is not a boolean.
  • Stated limit: a call in tail position as the last operand of and/or stays a tail call and is not checked itself, so deep mutual recursion through it keeps working (tested at depth 5M). A test of the result is what raises. Compared with other ports, g returning 5:
form this PR 912596b jit scheme go
(zf 5) 5 5 raises raises 5
(zf2 5) 5 5 raises raises 5
(zt1 5) raises yes raises raises raises
(zt2 5) raises yes raises raises raises
(zn1 5) 5 5 raises raises 5
(zn2 5) 5 5 raises raises 5
(zn3 5) 5 5 raises raises 5
(zn4 5) 5 5 raises raises 5
(zvp 5) raises 5 raises raises 5
(zit 5) raises yes raises raises raises
(zlm 5) 5 5 raises raises 5
(zfr 5) 5 5 raises raises 5
(zvr (/. X X) 5) 5 5 raises raises 5
(zpr1 5) raises 5 raises raises 5
(zca 5) raises true raises raises raises
(and true 5) raises 5 raises raises 5
(or false 5) raises 5 raises raises 5
(and true (zg 5)) raises 5 raises raises 5
(if (zg 5) a b) raises a raises raises raises
(and (zg 5) true) raises true raises raises true
(or (zg 5) true) raises true raises raises true
(cond ((zg 5) a) (true b)) raises a raises raises b
(not (zg 5)) raises false raises false raises
(let X (zf 5) (+ X 1)) 6 6 raises raises 6

Narrower than shen-go, wider than Shen/JIT and Shen/Scheme.

  • (str "a") returns the quoted form (4adf578, a separate commit you can drop), matching the other ports.
  • Rejected with an error now: bad absvector lengths, out-of-range address->/<-address, non-symbol set/value, hash with a non-positive bound, unknown get-time options.
  • Fixed: a trapped or discarded compile no longer leaves state behind; (get-time unix) is Unix time; (eval-kl sym) gives the symbol; close is idempotent; a fasl replays a macro's set of an unchanged value.
  • Calling a function value from Lua: a compiled lambda given fewer arguments than it takes returns a partial, wherever the host got it. This replaces a boundary wrapper, which leaked. It costs nothing measurable. Edge: a function installed directly in shen.prims.F that returns nil, whose result is passed as a lambda's last argument, gets a partial. (thaw (/. X Y ...)) gives a partial again, and thaw of a one-parameter lambda gives the lambda itself (912596b ran its body with nil). From the host only, raw over-application add(1, 2, 3) gives 3, where 912596b returned a function. The nil edge can't be reached through the Lua bridge, which turns nil into (); the reviewer tried.
  • Two evaluations of a closed lambda are the same function (= can be true).
  • Adaptive GC: it stops once the host sets its own pause. Fasl and signature cache formats bump (SHENFASL9, SHENSIG2).
  • Kept on purpose: too few arguments on a direct KL call returns a closure, and a symbol in a variable can be called.

Bugs found and fixed since 32dbc61

  • Tail calls through the last operand of and/or were lost (strict-test regression): in defuns, freeze and lambda bodies, and top-level lambdas.
  • Count sinking compiled a branch twice and read an undeclared local as nil, which gave a silent wrong answer. Fixed by rolling back every piece of compiler state a discarded attempt left. codegen_diff_spec now compares 495 shapes plus fuzz with each optimisation off, and scans every compiled chunk's bytecode for undeclared globals.
  • CURnP looked a symbol head up again after its last argument; non-tail calls only.
  • A predicate with too few arguments skipped the boolean check.
  • A host's bare-chunk function was taken for a compiled lambda.
  • A host's yield through a stack segment gained trailing nils, and a PUC hook leaked into the next job.
  • PROVEN facts survived a failed compile.
  • Fasl replay dropped a macro's set.
  • make certify exited 0 whatever the suite did, and its Counters line always read 0. Now it fails on a failure, an early stop, or a count other than the kernel's (154 for 43.2, kept per kernel version). Shown: one test broken gives make rc 2.
  • CI didn't see a native typechecker fallback and never ran the golden corpus. Both now run, and a spec missing from the runner's list fails the run (system_s_sigf_spec had never run).
  • Lua 5.5 did not boot, on 912596b too.

Verification on 5a0228d

Code SHA cc2a162; head 5a0228d differs from it only in CHANGELOG.md, and the specs were re-run on 5a0228d (2373/0).

  • Specs: 2373 pass, 0 fail, 38 specs. make certify and make golden run the way CI runs them: rc 0, Counters: passed=154 failed=0 expected=154, golden 27/27.
  • Kernel 154/0 and extensions 123/0, on the native and the legacy Prolog engine, first run and warm. Kernel 154/0 with SHEN_TYPECHECK_NATIVE=require as well.
  • The native engines really run, counted through a debug hook over the golden corpus: 8,670,175 calls into prolog_engine, 86,624 into typecheck_native, and none under legacy. Golden inference counts match 912596b's, entry for entry, on both engines (y-combinator 431741, einstein 33542).
  • THORN, exact inference counts and proof hashes: steam 5088693 bbac2f1d8d3e (also on the legacy engine), thm13 62098917 60f4dc46aa15, rcc10 47334656 with verdict false, thm55 475496076 aaf43ed5db99 (4254 G, low mode).
  • Contract examples 140/17; 912596b stopped at 82/14. benchmark-coverage passes on both engines. hello_embed works, and the single-file bundle boots from another directory.
  • Programs: 8 example selftests and config_check, bifrost 7/7, a 38-program sweep with no "is not a boolean" error, fuzz seeds 1-12 identical to 912596b, 31 numeric probes (including integers above 2^53 and overflow) identical to 912596b, and 64 tail-call shapes at depth 5M (top-level ones included) all pass.
  • PUC Lua 5.1.5, 5.4.9 and 5.5.1: kernel 154/0, hello_embed, and the error_robustness, calls, accumulator, embed_values and seg_pool specs all pass. Under PUC the full spec runs fail exactly the same specs as on 912596b (FFI, the cache, deep recursion), with none new. 912596b did not boot on 5.5.

Known and not in this PR

  • (shen.mod 5 0) hangs on all five ports tested. (shen.mod 7 -3) hangs here, on Shen/Scheme and on shen-go; Shen/JIT raises.
  • The extensions suite and benchmark-coverage need extensions/ and benchmarks/ from shen-sources, which aren't vendored; CI can't run them. Here they were run against Shen/JIT's 43.2 copies.
  • 5 bench scripts need ../cl-source or tla.shen and don't run.
  • Not tested: x86-64 performance; Ubuntu's LuaJIT, beyond CI.

🤖 Generated with Claude Code

pyrex41 and others added 30 commits November 5, 2026 15:01
(absvector -1) built a table whose length word disagreed with its slots;
printing it raised "0 is not a valid index" from outside the program's
handler and ended the process. (absvector 1.5) and (absvector "1") were
also accepted, and a non-number surfaced a raw Lua arithmetic error. All
now raise "<x> is not a valid vector length", the wording Shen/Scheme uses.

Cost, measured warm on 5M (absvector 2) calls, instructions retired:
14.62B, 14.59B before; 14.79B, 14.98B after (+1.2..2.7%, CPU within noise).
No kernel path allocates this way in a loop: `vector` has its own native
override and w/vec.shen calls absvector once.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A form the compiler rejects, such as (eval-kl [defun f [x] [lambda]]) or a
bare [lambda], left CTX and the other module-level compile state set. Every
later compile then emitted KC[i] or NT references into a chunk declaring
neither, so the session died with "attempt to index global 'KC'" on the next
form that hoists a control form, even though the first error had been trapped.
A rejected defun with a new arity also changed C.ARITY for later calls to the
surviving old definition.

This one bug produced 11 of the 35 contract failures (fn-applied, fn-partial,
get-*, vector-real, vector-negative, vector-symbol-length,
plus-vector-long-zero, tuple-parts): 122 -> 133 passing, same run.

Cost, 1500 defines with fasl off (w/compile.shen), instructions retired:
30.00B, 30.67B before; 30.35B, 29.98B after (no difference beyond noise).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The native variable? delegated non-symbols to the kernel version, whose
(str V) raises inside a trap-error for every cons and (). jit.p put 27-33%
of a cold load of 1500 defines in that pcall (C code: LuaJIT's error path
through the system unwinder). Every non-symbol answers false either way.

Instructions retired, billions (dd24cbb -> this), two runs or more each:
  1500 defines, cold fasl   28.5 -> 14.0   (cpu 2.54 -> 1.83)
  defines+call, cold fasl   28.8 -> 15.1
  kernel suite, warm fasl   22.1 -> 15.9   (154/0)
  kernel suite, cold fasl   76.7 -> 62.0
  extensions, warm fasl     2.06 -> 1.85   (123/0)
  startup, warm             0.75 -> 0.60
Specs 1797/0 before the new variable? cases.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(set "x" 1) bound the symbol x and (set 1 2) bound a variable called "1", so
strings aliased symbols and a mistake passed silently. Shen/Scheme, Shen/JIT
and the contract example glob-set-non-symbol raise. This path is the fallback:
compiled (value sym) is VALUE_S and does not reach it. No hot-path change, not
re-measured.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
P.eval returned the global's value when a bare symbol was bound, so after
(set x 7), (eval x) and (eval-kl x) gave 7, while a compiled x gave x.
Shen/Scheme, Shen/JIT and Shen/C all give x. Not covered by the contract
examples; found while checking them. Not on a hot path, not measured.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(hash 1 0) and (hash 1 -1) never returned: the kernel's shen.mod loops for a
non-symbol key. (hash a 0) returned NaN and (hash a 0.5) returned 1. Shen/JIT
raises for all of these, Shen/Scheme for 0 and 0.5. The kernel only calls hash
with a dict capacity, which is positive. Not covered by the contract examples.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A second close raised Lua's "attempt to use a closed file" with a source path
in the message; Shen/Scheme and Shen/JIT return []. read-byte after EOF on a
closed stream returned -1 instead of raising, and (close 1) returned [] where
Shen/Scheme raises. Swapping the stream's functions on close keeps the read and
write paths free of any check.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(get-time unix) returned seconds since boot, e.g. 2, where Shen/Scheme and
Shen/JIT return about 1793913049, because the unix and real arms shared one
line. Any other argument took the same arm, so (get-time sideways) returned a
number. real stays seconds since boot.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(str "a") returned a; shen-cl (`format nil "~S"`), Shen/Scheme, shen-go, Shen/C
and Shen/JIT all return "a" with its quotes, and the contract example
str-of-string expects that. The kernel's printer handles strings itself
(shen.str->str) and never reaches this path, so ~A and ~S are unchanged, and
concat of atoms is unchanged. (concat "a" b) now gives the symbol "a"b, as on
Shen/Scheme, rather than ab.

User-visible: a program that relied on (str S) returning S bare will now see
the quotes. test/utf8_strings_spec.lua locked the old result and is updated.
Specs 1869/0, kernel 154/0, extensions 123/0 on this commit.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
shen.record-kl sets shen.*userdefs* at every define and the fasl recorder
stored the full list each time: for 1500 defines 1.1M list cells, 12.4 of the
14.3 MB file, all parsed back into conses on a hit. New "h" record: the head
pushed onto the global's value. A set to the same value records nothing.
FASL_FORMAT bumped to SHENFASL8.

Instructions retired, billions, 2+ runs each (variable? fix -> this):
  1500 defines, warm fasl  3.36 -> 0.95   footprint 155 -> 32 MB
  1500 defines, cold fasl  14.0 -> 12.3   footprint 196 -> 139 MB
  kernel suite, warm fasl  15.9 -> 15.1   (154/0)
  typed load, warm fasl     (dd24cbb 6.3) -> 2.3
Extensions 123/0, specs 1810/0.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Reading went through the KL combinators per character: about 20,000
instructions per input byte, 4.3G of the 13.8G a cold 1500 defines spent once
variable? stopped throwing. sx_step now dispatches on the token's first code:
whitespace runs, brackets, closers, symbols (with the lookahead that leaves
signed numbers alone), strings without c# or non-ASCII, runs of at most 15
digits not followed by . or e. Everything else takes the kernel's rules as
before. reader_native_spec's differential against the KL gains 111 edge cases
aimed at these paths; 256 inputs compared, all equal.

Instructions retired, billions, 2 runs each:
  shen.<s-exprs> self time, 1500 defines   4.33 -> 0.19
  1500 defines, cold fasl                 12.3 -> 8.2  (cpu 1.26 -> 0.93)
Specs and suites run together with the next commit: specs 1921/0, kernel 154/0,
extensions 123/0.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
macroexpand makes one walk per macro (8 at boot) over each form; nearly always
nothing changes, yet the kernel's walk rebuilt the whole tree through map and
macroexpand-h then compared it to the old one cell by cell. jit.p on macroexpand
alone: 56% in map plus cons, 24% GC. Native walk: same macro calls in the same
order, allocation only from the first changed child, so the comparison ends on
identity. Forms are immutable, so the sharing is not observable.

Instructions retired, billions, 2 runs each (previous commit -> this):
  macroexpand self time, 1500 defines      3.15 -> 0.82
  1500 defines, cold fasl                  8.2 -> 5.6   (cpu 0.93 -> 0.70)
Specs 1921/0, kernel 154/0, extensions 123/0.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(address-> v 5 x) grew the table silently, (address-> v -1 x) overwrote the
length word, and (<-address v -1) read it back as a value. Shen/Scheme and
Shen/JIT raise for all of these. Reads past the end already raised. Kernel 154/0
and extensions 123/0 pass, so nothing in the kernel stores out of range.

One known gap stays: a cons is {h,t}, so (<-address (cons 1 2) 0) still returns
the tail, because rejecting it needs a metatable check on every access.

Cost, warm, instructions retired, A then B:
vec    13.09B 13.22B -> 13.28B 13.28B (+0.4..1.4%)
prolog 22.13B 20.63B -> 20.48B 22.66B (no difference beyond noise)
typed   6.18B  6.15B ->  6.15B  6.06B (no difference beyond noise)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The pattern compiler tests (cons? V) and then calls HD/TL, which test it
again. In code a test dominates, emit E[1] / E[2] instead. Facts come from
if, cond and (and ...) tests over chains rooted at a bound variable. A cons
stays a cons and a Lua name is bound once; the one reassignment, of a
self-tail loop's params, is followed by goto, which re-tests.

nrev was bimodal: 9 of 12 runs at 285-289 billion instructions, 3 of 12 at
184-188. In the slow mode, jit.v shows TSTEP rooted as a "return" trace, with
TLOOP's body left to the interpreter after recording stopped at the boot-time
root trace of HD/is_cons. With no call on the hot path, 12 of 12 runs give
179-183.

Instructions retired, billions, warm, private kernel cache; two runs each:
  nrev  285-289 (or 184-188)  -> 179-183   CPU 17.9 -> 14.7 s
  hof   48.2 48.9             -> 42.0 41.9
  str   3.77 4.19             -> 3.53 3.46
  extensions 2.12 2.08        -> 2.04 2.03
  fib, loop, vec, typed, compile, startup, prolog, kernel: within noise
make test 1797/0, kernel 154/0, extensions 123/0.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(stlib.initialise-datatypes) ran the rule compiler for its two datatypes on every
start: 300M of a 600M warm boot. The cold boot now records the phase with the
fasl recorder and stores the stream in the sidecar, SHENSIG2. A boot with
SHEN_FASL=off stores none and runs the phase, and boot_cache_spec's fingerprint
still equals an uncached boot in every cache configuration.

Instructions retired, millions, warm, 3 runs each:
  startup  ~590 -> ~490   (stage delta: datatypes 300 -> 10; footprint 21 -> 20.6 MB)
Specs 1921/0, kernel 154/0, extensions 123/0.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Every boot parsed types.kl (which has no declares on 43.1) and init.kl with
R.read_all to fill the native signature table: 65M of 480M instructions, for a
table only the native typechecker reads. The files are still read at install,
as the working directory can change, and parsed at the first typecheck. A
signature declared or destroyed before then stands; typecheck_lazy_spec pins it.

Instructions retired, millions, warm, 3 runs each:
  startup  ~490 -> ~410    footprint 20.5 -> 20.4 MB
Programs that typecheck pay the same, later. Specs 1927/0, kernel 154/0,
extensions 123/0.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Finishing a cycle sooner leaves less floating garbage in the estimate that
sets the next threshold. Peak footprint, MB: kernel 186 -> 163, nrev 40-56 ->
37, extensions 33 -> 29; hof 126 -> 115-122. startup 23 -> 25, vec 25 -> 27.
Instructions and CPU within noise (nrev 181.2 either way).
Instructions-only numbers would show nothing here: this is a memory change.

Cost: each GC step does more work at once, so individual steps are longer.
SHEN_GC=400,200 restores the previous setting.

Tested and not adopted, larger pauses (with stepmul 1000):
  600: nrev 176 B, 12.0 s, 56 MB; hof 232 MB; kernel 263 MB
  800: nrev 174 B, 11.2 s, 74 MB; hof 1133 MB; kernel 281 MB
make test 1807/0, kernel 154/0, extensions 123/0.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A curried chain through a defun of arity n calls PARTIAL n-1 times. Each
partial's call copied its arguments into two tables and looped over them. jit.p
on the kernel suite put 8% of samples there, and more when those loops did not
trace.

Billions of instructions, warm, base dd24cbb -> this:
kernel suite 22.3 -> 19.3 (CPU 2.6 -> 1.8 s, footprint 187 -> 170 MB, 154/0),
typed load 6.1 -> 6.0; hof, fib, compile unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ists

Attribution on the warm kernel suite: shen.variancy 12.2G of 15.1G, 174 calls at
70M each. It is declare's System S query on the kernel's Prolog for the function's
old type, and all it can do is warn that the type changed. A replay of a name with
no signature now skips it. With a signature, declare runs in full, so the warning
still appears, cli_spec pins it cold, warm and with SHEN_FASL=off. A user datatype
rule giving (fn F) a type for a function with no signature could have warned on
the cold run and will not on a hit: that case alone differs.

Instructions retired, billions, warm fasl, 2-3 runs each:
  kernel suite   15.1 -> 3.6    footprint 185 -> 82 MB   (154/0)
  typed load      2.3 -> 1.4    footprint 78 -> 54 MB
  extensions     1.46 -> 1.39   (123/0)
Specs 1932/0 outside cli_spec, cli_spec 86/0 after fixing the new test itself.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…mbdas

Generated code for (F A B) with F a variable was APP(APP(F, A), B): two vararg
dispatches, plus a closure from the first when F took both. Now:
- APP1..4 replace APP where the count is known.
- ((F a) b) is CURn(F, a, b), a direct call when F's arity is n. A last argument
  that is not an atom uses CURnP/PREn, which run any partial application before
  evaluating it, so order and errors are those of the nested form.
- A symbol in the function position resolves without a partial.
- A one-parameter lambda writes no FA entry. That was a new key in a weak table
  per closure (44% of APP's samples on hof). APP recovers its arity from
  debug.getinfo, memoised, on the path where the count is not 1. Lua 5.1 has no
  nparams and keeps MKFUN.
- The helpers are vararg functions so they never anchor a root trace: as plain
  functions APP1 became a shared root, every loop through it linked into it, and
  hof's instructions ranged 11.7-20.4 B run to run.

Billions of instructions, warm, previous commit -> this:
hof 48.1 -> 49.9 instructions, CPU 4.4 -> 3.2 s, footprint 120 -> 54 MB (a win in
time and memory, not in instructions: it removes cache and GC work);
kernel suite 19.3 -> 18.1 (CPU 1.9 -> 1.56 s, footprint 170 -> 156 MB, 154/0);
typed 6.0 -> 5.9; compile warm 3.32 -> 3.27, cold 25.1 -> 26.1; fib unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(lambda X (lambda Y B)), up to 4 deep, is one function: a call with every
argument allocates nothing where it used to allocate a closure per step. A partial
goes through PARTIAL, as for a defun. A lambda reading no outer local is built once
in the defun's KC table. FNEW is a bytecode LuaJIT cannot trace, and jit.v showed
every trace through fold aborting on it. Two evaluations of a closed lambda are now
the same function, the only visible change.

Billions of instructions, warm, previous commit -> this:
hof 49.9 -> 13.5 (CPU 3.2 -> 0.71 s);
kernel suite 18.1 -> 18.5 (CPU unchanged at 1.58 s, 154/0), typed 5.9 -> 6.07,
within or near run-to-run noise; compile warm 3.27 -> 3.32, cold 26.1 -> 25.6;
fib unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…it runs

ADD, HD, APPn, setmetatable and the rest were chunk globals: a hash lookup in
ENV and an identity guard per use. A never-reassigned local is an immutable
upvalue, and a trace folds a function one to a constant. Only names a body
mentions are bound, and wide mode binds none, as it budgets its upvalues without
them. Every non-exhaustive define also read F["shen.f-error"] on entry.

Billions of instructions, warm, previous commit -> this:
fib 5.0 -> 4.3; hof 13.5 -> 10.2; typed 6.07 -> 5.9; kernel suite 18.5 -> 18.0
(CPU 1.58 -> 1.68 s, within noise; 154/0); compile warm 3.32 -> 3.38, cold
25.6 -> 23.9. Startup 0.75 -> 0.80.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Peak footprint was pause 400 on top of live data, not retained data: the kernel
suite peaked at 186 MB with 36 MB live after a full collection. SHEN_GC=200 on
dd24cbb gave 97 MB at the same instructions, but cost nrev dearly, so a fixed
pause is the wrong knob. A finaliser that re-arms each cycle now sets the pause to
reach live + 48 MB, clamped to 150..400, so a small heap keeps 400. An explicit
SHEN_GC stays fixed, SHEN_GC=off still leaves the host's settings.

Instructions retired, billions, and peak footprint, 2 runs each (before -> after):
  kernel suite, cold   61.5, 322 MB -> 54.4-67, 150 MB   (154/0; instructions noisy)
  kernel suite, warm    3.6,  82 MB -> 3.6,  78 MB
  1500 defines, cold    5.4, 116 MB -> 5.3,  93 MB
  closures (hof)       47.8, 127 MB -> 48-50, 70 MB
  vec, fib, str, prolog, typed: unchanged
  3M-cell live list plus churn: 13.8, 1276 MB -> 17.7, 567 MB   (the cost: +28%)
Specs 1941/0, kernel 154/0, extensions 123/0, examples/hello_embed.lua exits 0.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…l call; calls spec

A counting defun wrote SD[1] on entry and on every return, including on the clauses
that make no call, half of all calls in a tree recursion. A defun's leading tests,
built from primitives that change nothing, now run uncounted, and each branch opens
the count if it needs it. A segment restarts the function from the top, which only
repeats those tests. A redefined primitive with side effects in such a test would
run twice at a segment boundary. Loop-lowered and TRMC defuns are unchanged.

Also: no uncurrying in wide mode, which budgets each closure's upvalues (fixes
datatype_consume_spec's 64-premise rule, broken since 5d6b380).

test/calls_spec.lua covers every application shape, redefinition with arity
change, evaluation order through CURnP, proper tail calls through each helper, deep
recursion across segments and host functions. It passes on dd24cbb as well (36/1
there before its expectation for a let-bound (fn f) was corrected to what dd24cbb
does).

Billions of instructions, warm, previous commit -> this:
fib 4.3 -> 3.65; hof 10.2 -> 10.1; typed 5.9 -> 5.85; kernel suite 18.0 -> 18.0
(154/0); compile warm 3.38 -> 3.25, cold 23.9 -> 25.0 (three runs of 24.9-25.1; dd24cbb
24.9-26.3 on the same load).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A Prolog query deeper than 96 counted frames opened a fresh coroutine every 96
frames, and each one regrew a stack from nothing. S.run now keeps up to 32
finished segments; the collector still shrinks an idle one's stack.

Warm, 5 runs, private SHEN_KERNEL_CACHE per tree, instructions retired (G):
  prolog before 20.56 20.64 20.79 22.36 22.75  cpu 1.8-2.1 s  peak 48 MB
  prolog after  15.86 15.91 15.91 16.05 17.22  cpu 0.9-1.1 s  peak 40 MB
Specs green with 17 checks added here; kernel 154/0; extensions 123/123.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
lazyderef and unify_core held loops that were inner loops of every trace
through a predicate. LuaJIT aborts on those, so traces were cut there or fell
back to the interpreter. One-step dereferences and unifications binding a
single variable now run with no loop; chains and two lists go out of line.

Warm, 5 runs, private SHEN_KERNEL_CACHE per tree, instructions retired (G):
  prolog before 15.86 15.91 15.91 16.05 17.22  cpu 0.9-1.1 s
  prolog after  14.25 14.72 14.92 15.10 16.28  cpu 0.8-1.05 s
Specs 1814/0; kernel 154/0; extensions 123/123.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A store into a table with a metatable makes the trace guard the type of the old
value in the slot. Vectors carry Vmt, which has no metamethods, so overwriting a
vector with values of another type left the trace at every element. rawset is
recorded with no such guard. Only the message for a non-table first argument
changes (rawset's "table expected" instead of "attempt to index").

Warm, private SHEN_KERNEL_CACHE per tree, 6 runs, instructions retired (G):
  vec    before 13.01 13.03 13.06 13.06 13.06 13.09  cpu 0.46-0.57 s
  vec    after   6.49  6.50  6.52  6.52  6.53  6.54  cpu 0.26-0.36 s
  kernel suite unchanged within noise (see final A/B: 22.3-22.5 vs 22.4-22.7)
Specs 1814/0; kernel 154/0; extensions 123/123.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Under dual-number LuaJIT (arm64) a self-tail loop's trace is specialised to the
types at the moment it is recorded. An accumulator that starts as an int and
leaves int32 range then fails the trace's guard on every iteration, running the
loop's head plus a side trace, about 10 times the cost of the loop. A self-tail
argument (+ V E), (+ E V), (- V E) or (* V E), for parameter V and non-literal
E, now always yields a double, so the trace is specialised to that.
Values and errors are the same. Constant steps stay ints, so counters and
indices are untouched. x86-64 LuaJIT has no int representation to leave.

Warm, private SHEN_KERNEL_CACHE per tree, instructions retired (G):
  vec   before 6.49 6.50 6.52 6.52 6.53 6.54  cpu 0.26-0.36 s
  vec   after  1.80 1.80 1.80 1.83            cpu 0.11-0.13 s
  fib 4.94-5.01 vs 4.94-5.03, loop 2.17-2.24 vs 2.16-2.21, typed 6.21 vs 6.26
Specs green with accumulator_spec added; kernel 154/0; extensions 123/123.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The kernel's shen.mod builds a list of doubled divisors and subtracts them, a
few thousand instructions per call. Non-negative integers below 2^53 now take
N % D, which is what that computes; everything else keeps the kernel's path.
This mostly helps programs calling shen.mod directly, such as the str workload,
since hash already had its own native path. It is not a string change.

Warm, private SHEN_KERNEL_CACHE per tree, instructions retired (G):
  str  before 3.77 3.80 4.03              cpu 0.28-0.35 s  peak 40 MB
  str  after  1.36 1.42 1.43 1.47         cpu 0.10-0.14 s  peak 25-27 MB
The string operations alone (no shen.mod or str, same counts) were already 1.36 G.
Specs green; kernel 154/0; extensions 123/123.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
pyrex41 and others added 6 commits November 6, 2026 01:37
A cons has a [1] too, and a number there read as a length: (address-> (cons 5 [])
3 8) grew the cell instead of raising. The check is now on the metatable, which also
removes the type test on the length. vec, private caches, 4 runs each: 1.89 G
(1.89-1.96) before, 1.88 G (1.85-1.98) after.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(if 1 a b) took the then branch with no error, since Lua treats anything but nil and
false as true; shen-go, shen-cl, Shen/Scheme and Shen/JIT raise. The test of if and
cond and both operands of and/or are now checked where the compiler emits them. A form
that can only give a boolean goes unchecked, and anything else is BC(form).

The last operand of and/or is in tail position: the check goes to the leaves of its
control forms, and a call to the function being compiled stays a tail call, so a loop
written (and (p X) (loop Xs)) keeps its frame. Its final value reaches whoever tests the
result. A tail call to any other function there is no longer a tail call.

(not 1) is unchanged here: it compiles to Lua's not and gives false.

Replaces the divergence pinned in error_robustness_spec. Costs: INTEGRATION.md.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
08c6adf kept this only for a call to the function being compiled, so mutual
recursion through and/or grew the stack: (ev? 5000000) with ev? and od? calling each
other as (and (number? N) (od? (- N 1))) raised at 10000 stack segments, where
912596b, Shen/JIT and Shen/Scheme give true. In a defun's tail a call there is now
left unchecked and its value is the function's result, which a test of it checks. In a
control form in value position or a top-level form nothing is a tail call, so a call
there is still checked. Stated gap: a caller that uses (and A (g X))'s value
without testing it can see a non-boolean from g.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…th 5M

The whole body of a freeze or a lambda compiled as a control form in value position,
so a tail call there was still checked and grew the stack. It is in tail position, so
it stays a tail call too. error_robustness_spec now covers mutual tail recursion at
depth 5M through and, or, and if, cond, let, do, and, or, freeze and lambda inside them,
and through a variable; on 32dbc61 it fails. It also pins the cases left unchecked, on
purpose: Shen/JIT and Shen/Scheme raise on those, shen-go gives the value.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@pyrex41 pyrex41 left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Summary

The boot, fasl, boolean-test, cons-fact, count-sinking, and closed-lambda work is the right shape, and most of the conformance histories in the PR body are actually forbidden by the code. Four fast paths are wrong. A fifth finding from the first pass, that unify_core overflows by skipping seg_depth when the terms are identical, does not hold: return thawH(cont) is a tail call, so that frame does not accumulate. The old counter existed because the thaw was not in tail position.

Panel: Linus Torvalds (do not merge yet), Joel Armstrong (failure isolation on the segment pool and on PROVEN), Kyle Kingsbury (one live violated history, the symbol-head CURnP re-lookup; the other claimed properties hold in the code, several of them untested).

Issue counts by severity

  • bugs: 4
  • suggestions: 1
  • nits: 0

Comment thread prims.lua Outdated
.. "end, function(g, " .. plist .. ", r, an, ...)\n"
.. " if r == D then\n"
.. " if type(g) == 'function' then return g(" .. plist .. ", an) end\n"
.. " return F[g.name](" .. plist .. ", an)\n"

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[bug] Exact-arity CURnP looks a symbol up again after the last argument has run. PREn (prims.lua:2999) already resolved F[g.name] and then threw that function away, returning the direct-call sentinel. CURnP then does F[g.name](a1..an). Nested APP, the path taken when the arity is not n, closes the partial over the function object before an is evaluated.

(let F add2 (F 1 (do (define add2 X Y -> 0) 2))) must yield 3. This path yields 0. calls_spec only covers order-free arguments, so it does not lock the case. The PR text names this re-lookup; it is still the wrong function.

Suggestion: Have PREn return the function it just resolved, and call that object. Do not read F[g.name] again after an.

Comment thread compiler.lua Outdated
if form == true or form == false then return true end
if not is_cons(form) then return false end
local h = car(form)
return is_symbol(h) and not env[h.name] and BOOLEAN_FORMS[h.name] == true

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[bug] gives_boolean is "the head symbol is in this set". It does not look at arity. (if (= 1) a b) and (if (cons?) a b) compile to a partial function with no BC wrap. A function is truthy in Lua, so the test takes the then-branch and never raises. Saturated (= a b) is fine, and and/or check their own operands. This is the check the patch exists to add, skipped on the calls that do not return a boolean.

Suggestion: Skip BC only when the call has the primitive's real arity (and, for and/or/not, the shape you actually inline). Otherwise wrap it.

Comment thread prims.lua Outdated
r = false
if getinfo then
local i = getinfo(f, "uS")
if i.nparams == 1 and not i.isvararg and i.what ~= "main" then

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[bug] Dropping MKFUN for one-parameter lambdas is fine. Recovering them with debug.getinfo is not. Any one-parameter, non-vararg Lua function whose chunk name does not start with @ or = is treated as a Shen lambda of arity 1: zero arguments become a partial, extra arguments are applied to the result. That includes load / loadstring of host code on LuaJIT and PUC Lua 5.2+, which this tree runs. Nothing at creation time records the fact, and RAW_LAMBDA then freezes the guess. On PUC 5.1 nparams is absent, so the guess stays false and the same source behaves differently per host.

Suggestion: Mark the closure when you build it (a weak table set from lam1, or keep MKFUN). Do not sniff source names.

Comment thread runtime.lua
local pool, npool = {}, 0
local POOL_MAX = 32
local function seg_body(f, ...)
return seg_body(co_yield(DONE, pcall(f, ...)))

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[bug] Reusing a finished coroutine is the right cost model. Two things about this wrapper are wrong.

pcall around the body keeps the coroutine alive across an error, and PUC Lua 5.1 cannot yield across pcall. LuaJIT and Lua 5.2+ can. A host yield inside a deep segment on 5.1 becomes attempt to yield across C-call boundary, which this pcall turns into a failed job, and seg_done then pools the coroutine as if the job finished (runtime.lua:368). The old runner resumed f directly and forwarded yields.

Pooling also keeps coroutine-local state. A debug.sethook installed on coroutine.running() inside a segment is still installed on the next S.run, and the one after that. This tree does not call sethook; an embedder that does now shares that hook across jobs. A fresh coroutine did not.

Suggestion: On 5.1, do not pcall the body; allocate a fresh coroutine after an error. Before pooling anywhere, drop the hook and any other coroutine-local state. Pooling the success path is enough.

Comment thread compiler.lua
for k in pairs(facts) do
if not PROVEN[k] then PROVEN[k] = true; added[#added + 1] = k end
end
local r = f(...)

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[suggestion] with_cons_facts records keys in the process-global PROVEN and clears them only after f returns. atomic_compile does not snapshot PROVEN. An error while compiling the branch leaves the key set. The key is a gen() name and is not issued again, so a later defun does not miscompile. The table still grows by one entry per failed cons-guarded compile, and it is the one mutated local outside the restore list.

Suggestion: Clear the keys you added if f errors, or put the previous table back on the atomic_compile failure path.

pyrex41 and others added 20 commits November 6, 2026 12:27
With F holding a symbol and a last argument that rebinds that name, (+ 0 (F 1 (do
(rebind) 2))) called the new definition; 912596b gives the old one's result. A tail
call never reached CURnP: a counting defun hoists its arguments, so it compiled as
APP1(APP1(F, 1), arg) and settled the function at the first application. Non-tail
calls compile as CUR2P(F, 1, PRE2(F, 1), arg), where CURnP looked the name up again
after the argument. PREn now hands over the function it found.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
gives_boolean looked only at the head, so (if (cons?) a b) and at KL level (if (= 1)
a b) skipped the check and took the then branch on the closure. shen-go and Shen/Scheme
raise. The check is skipped now only for a saturated call.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
is_raw_lambda took any chunk with a bare name as compiled, and a host's bare loadstring
has one. So APP on a host's unary function with 2 arguments errored (912596b ignores
the extra), with 0 returned a partial, and a curried one with 2 curried. load_chunk now
records each chunk's source name and only those count. A lookup per chunk at load and on
APP's other-than-one-argument path: nothing per closure, unlike a mark at creation.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
seg_done packed fixed slots on a host's yield, so a yield of one value reached the
host with trailing nils. Exactly the values yielded now pass through. On PUC Lua a hook
set during a job stayed on the pooled coroutine and the next job inherited it, so a
coroutine with a hook is not reused. seg_pool_spec covers both and runs on LuaJIT and
PUC 5.1; the yield case runs on LuaJIT, since PUC 5.1 cannot yield across a segment's
pcall on 912596b either.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
with_cons_facts undoes its keys on return, which an error skips. gen()'s counter is
monotonic, so a stale key cannot match later code: this is growth, not a miscompile. A
failed compile now clears them, with the rest of its state.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(not 5) compiled to Lua's not and gave false, while the primitive raises. Shen/JIT and
shen-go raise, Shen/Scheme gives false. An argument that can only give a boolean is
unchecked, as for the other tests. Costs: INTEGRATION.md.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
run-kernel-tests.lua exited 0 whatever the suite did: its only os.exit(1) was for a
harness that would not load. A failed test, a suite stopping midway and a truncated
suite all left make certify and CI green. Its Counters line read the harness's globals
after kerneltests.shen's final (reset), so it always said 0. It now counts each test as
it ends and exits 1 unless the suite finished, none failed and exactly 154 ran; a kernel
that adds tests needs that number bumped. A real run with one test broken gives
make rc 2, and 912596b's real count is 154 passed, 0 failed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Nothing in CI noticed the native typechecker falling back to the kernel's, since a
fallback only prints to stderr and the suite still passes: certify now runs with
SHEN_TYPECHECK_NATIVE=require. bench/golden_typecheck.lua could not run from a checkout;
it now uses tests/ and also checks the y-combinator's 431741 inferences, as make golden,
in CI. scripts/run-tests.lua has a fixed list, and test/system_s_sigf_spec.lua was not
on it: it is now, and a spec missing from the list fails the run.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Count sinking compiles a branch, and when a test turns out to make a counted call,
compiles it again. The first attempt's CCACHE entries named locals declared only in the
discarded text, so the second read an undeclared global: nil, with no error.
(pick 0 [0 1] 0) gave b, and an hd of a non-cons returned instead of raising.

It takes all of: a lambda capturing a parameter, which leaves the defun without a
self-tail loop, the only case that sinks at its top; a test with a control form in
value position whose tail is a call outside LEAF, such as not, which makes the helper's
call counted; and a branch that hoists an accessor chain to a local.

compile_attempt now rolls back CCACHE, the local budget, KC bodies, the counted-call
flags and counters, the hoist list and F references. try_trap_opt uses it as well: it
compiles its default before it knows the fast path applies. No repro for that site.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
leave_as_tail_call tested for a defun, so a top-level lambda's last operand of and/or was
checked, and hit the stack limit: (set top (/. N (or (= N 0) ((value top) (- N 1))))) at
3M. 912596b, Shen/JIT, Shen/Scheme and shen-go give true. The whole body of a lambda or a
freeze now counts as a function's tail wherever it sits.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ments

Uncurrying changed what a host could do with a function it got back:
shen.eval("(/. X Y (+ X Y))")(1)(2) raised, and shen.eval("(/. X (/. Y (cons X Y)))")(1)
returned a malformed cons with no error. Compiled functions take exactly their arity,
and a host does not know it. shen.eval, shen.call, shen.fn, shen.totable and the interop
marshalers now give a wrapper through APP, and take it back as the function it stands
for, so internal calls cost nothing. A host's own function crosses as itself.

Stated limits, in the README and CHANGELOG: a function inside a list or vector from
shen.eval is the raw one, applied with the new shen.apply; shen.prims.F[name] is raw too.
The wrapper is a different Lua value from the raw function: the API hands out only one of
them, but Shen code holding the raw one and a host comparing with the wrapper would differ.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The recorder skipped a macro-time (set ...) when the global already held that value at
compile time, so a session replaying the fasl without it saw no value at all, where an
uncached run has it. Compiled with (set cfg 5) already done, then replayed fresh: unset.
It now records the value once per fasl, and skips only repeats.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A kernel with no entry fails and names the table to update, rather than passing or
failing on a stale 154.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
codegen_diff_spec runs 495 shapes built to reach count sinking's second compile and a
fixed fuzz program (scripts/codegen-fuzz.py, from the independent review) with each
optimisation switch off, and compares results. It also scans the bytecode of every chunk
compiled for the kernel, stdlib and both programs for a global that is not in the compiled
environment: a local whose declaration was lost reads as nil, with no error. With
compile_attempt's child cache removed, which is the original bug, it fails on both
counts. About 13 s.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Lua 5.5 removed collectgarbage's setpause and setstepmul, so require failed at boot, on
912596b too, though the README claims 5.5. Three deep-recursion checks in calls_spec now
run on LuaJIT only: PUC raises C stack overflow on those at 912596b as well.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
It left a function raw on every route it did not cover: one fetched with shen.value,
one Shen passes to a host callback, and one a host puts inside a list compared with =,
which failed. The fix belongs in the function itself: next commit.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
An uncurried lambda called raw with too few arguments ran its body with nil, so a host
calling a function value it got from anywhere (shen.eval, shen.value, a callback, a list)
got a wrong result or an error, and so did (thaw (/. X Y ...)). Each compiled lambda now
tests its last parameter on entry: nil is never a Shen value, so nil there means
arguments are missing, and it returns the partial application over the ones it got
(UPART). Compiled calls always pass every argument or go through APP.

Checked that nil is never a Shen value in practice by logging every firing of the guard:
0 across 37 specs, the kernel suite, extensions, 9 example programs, THORN steam and the
contract file; 18, all deliberate, in embed_values_spec. A host function that returns nil
into Shen, passed as a lambda's last argument, would now get a partial.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…nil edge

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@pyrex41
pyrex41 merged commit 38ece2d into main Nov 7, 2026
2 checks passed
@pyrex41 pyrex41 mentioned this pull request Nov 7, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant