Repository navigation
Conversation
(absvector -1) built a table whose length word disagreed with its slots; printing it raised "0 is not a valid index" from outside the program's handler and ended the process. (absvector 1.5) and (absvector "1") were also accepted, and a non-number surfaced a raw Lua arithmetic error. All now raise "<x> is not a valid vector length", the wording Shen/Scheme uses. Cost, measured warm on 5M (absvector 2) calls, instructions retired: 14.62B, 14.59B before; 14.79B, 14.98B after (+1.2..2.7%, CPU within noise). No kernel path allocates this way in a loop: `vector` has its own native override and w/vec.shen calls absvector once. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A form the compiler rejects, such as (eval-kl [defun f [x] [lambda]]) or a bare [lambda], left CTX and the other module-level compile state set. Every later compile then emitted KC[i] or NT references into a chunk declaring neither, so the session died with "attempt to index global 'KC'" on the next form that hoists a control form, even though the first error had been trapped. A rejected defun with a new arity also changed C.ARITY for later calls to the surviving old definition. This one bug produced 11 of the 35 contract failures (fn-applied, fn-partial, get-*, vector-real, vector-negative, vector-symbol-length, plus-vector-long-zero, tuple-parts): 122 -> 133 passing, same run. Cost, 1500 defines with fasl off (w/compile.shen), instructions retired: 30.00B, 30.67B before; 30.35B, 29.98B after (no difference beyond noise). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The native variable? delegated non-symbols to the kernel version, whose (str V) raises inside a trap-error for every cons and (). jit.p put 27-33% of a cold load of 1500 defines in that pcall (C code: LuaJIT's error path through the system unwinder). Every non-symbol answers false either way. Instructions retired, billions (dd24cbb -> this), two runs or more each: 1500 defines, cold fasl 28.5 -> 14.0 (cpu 2.54 -> 1.83) defines+call, cold fasl 28.8 -> 15.1 kernel suite, warm fasl 22.1 -> 15.9 (154/0) kernel suite, cold fasl 76.7 -> 62.0 extensions, warm fasl 2.06 -> 1.85 (123/0) startup, warm 0.75 -> 0.60 Specs 1797/0 before the new variable? cases. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(set "x" 1) bound the symbol x and (set 1 2) bound a variable called "1", so strings aliased symbols and a mistake passed silently. Shen/Scheme, Shen/JIT and the contract example glob-set-non-symbol raise. This path is the fallback: compiled (value sym) is VALUE_S and does not reach it. No hot-path change, not re-measured. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
P.eval returned the global's value when a bare symbol was bound, so after (set x 7), (eval x) and (eval-kl x) gave 7, while a compiled x gave x. Shen/Scheme, Shen/JIT and Shen/C all give x. Not covered by the contract examples; found while checking them. Not on a hot path, not measured. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(hash 1 0) and (hash 1 -1) never returned: the kernel's shen.mod loops for a non-symbol key. (hash a 0) returned NaN and (hash a 0.5) returned 1. Shen/JIT raises for all of these, Shen/Scheme for 0 and 0.5. The kernel only calls hash with a dict capacity, which is positive. Not covered by the contract examples. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A second close raised Lua's "attempt to use a closed file" with a source path in the message; Shen/Scheme and Shen/JIT return []. read-byte after EOF on a closed stream returned -1 instead of raising, and (close 1) returned [] where Shen/Scheme raises. Swapping the stream's functions on close keeps the read and write paths free of any check. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(get-time unix) returned seconds since boot, e.g. 2, where Shen/Scheme and Shen/JIT return about 1793913049, because the unix and real arms shared one line. Any other argument took the same arm, so (get-time sideways) returned a number. real stays seconds since boot. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(str "a") returned a; shen-cl (`format nil "~S"`), Shen/Scheme, shen-go, Shen/C and Shen/JIT all return "a" with its quotes, and the contract example str-of-string expects that. The kernel's printer handles strings itself (shen.str->str) and never reaches this path, so ~A and ~S are unchanged, and concat of atoms is unchanged. (concat "a" b) now gives the symbol "a"b, as on Shen/Scheme, rather than ab. User-visible: a program that relied on (str S) returning S bare will now see the quotes. test/utf8_strings_spec.lua locked the old result and is updated. Specs 1869/0, kernel 154/0, extensions 123/0 on this commit. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
shen.record-kl sets shen.*userdefs* at every define and the fasl recorder stored the full list each time: for 1500 defines 1.1M list cells, 12.4 of the 14.3 MB file, all parsed back into conses on a hit. New "h" record: the head pushed onto the global's value. A set to the same value records nothing. FASL_FORMAT bumped to SHENFASL8. Instructions retired, billions, 2+ runs each (variable? fix -> this): 1500 defines, warm fasl 3.36 -> 0.95 footprint 155 -> 32 MB 1500 defines, cold fasl 14.0 -> 12.3 footprint 196 -> 139 MB kernel suite, warm fasl 15.9 -> 15.1 (154/0) typed load, warm fasl (dd24cbb 6.3) -> 2.3 Extensions 123/0, specs 1810/0. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Reading went through the KL combinators per character: about 20,000 instructions per input byte, 4.3G of the 13.8G a cold 1500 defines spent once variable? stopped throwing. sx_step now dispatches on the token's first code: whitespace runs, brackets, closers, symbols (with the lookahead that leaves signed numbers alone), strings without c# or non-ASCII, runs of at most 15 digits not followed by . or e. Everything else takes the kernel's rules as before. reader_native_spec's differential against the KL gains 111 edge cases aimed at these paths; 256 inputs compared, all equal. Instructions retired, billions, 2 runs each: shen.<s-exprs> self time, 1500 defines 4.33 -> 0.19 1500 defines, cold fasl 12.3 -> 8.2 (cpu 1.26 -> 0.93) Specs and suites run together with the next commit: specs 1921/0, kernel 154/0, extensions 123/0. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
macroexpand makes one walk per macro (8 at boot) over each form; nearly always nothing changes, yet the kernel's walk rebuilt the whole tree through map and macroexpand-h then compared it to the old one cell by cell. jit.p on macroexpand alone: 56% in map plus cons, 24% GC. Native walk: same macro calls in the same order, allocation only from the first changed child, so the comparison ends on identity. Forms are immutable, so the sharing is not observable. Instructions retired, billions, 2 runs each (previous commit -> this): macroexpand self time, 1500 defines 3.15 -> 0.82 1500 defines, cold fasl 8.2 -> 5.6 (cpu 0.93 -> 0.70) Specs 1921/0, kernel 154/0, extensions 123/0. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(address-> v 5 x) grew the table silently, (address-> v -1 x) overwrote the
length word, and (<-address v -1) read it back as a value. Shen/Scheme and
Shen/JIT raise for all of these. Reads past the end already raised. Kernel 154/0
and extensions 123/0 pass, so nothing in the kernel stores out of range.
One known gap stays: a cons is {h,t}, so (<-address (cons 1 2) 0) still returns
the tail, because rejecting it needs a metatable check on every access.
Cost, warm, instructions retired, A then B:
vec 13.09B 13.22B -> 13.28B 13.28B (+0.4..1.4%)
prolog 22.13B 20.63B -> 20.48B 22.66B (no difference beyond noise)
typed 6.18B 6.15B -> 6.15B 6.06B (no difference beyond noise)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The pattern compiler tests (cons? V) and then calls HD/TL, which test it again. In code a test dominates, emit E[1] / E[2] instead. Facts come from if, cond and (and ...) tests over chains rooted at a bound variable. A cons stays a cons and a Lua name is bound once; the one reassignment, of a self-tail loop's params, is followed by goto, which re-tests. nrev was bimodal: 9 of 12 runs at 285-289 billion instructions, 3 of 12 at 184-188. In the slow mode, jit.v shows TSTEP rooted as a "return" trace, with TLOOP's body left to the interpreter after recording stopped at the boot-time root trace of HD/is_cons. With no call on the hot path, 12 of 12 runs give 179-183. Instructions retired, billions, warm, private kernel cache; two runs each: nrev 285-289 (or 184-188) -> 179-183 CPU 17.9 -> 14.7 s hof 48.2 48.9 -> 42.0 41.9 str 3.77 4.19 -> 3.53 3.46 extensions 2.12 2.08 -> 2.04 2.03 fib, loop, vec, typed, compile, startup, prolog, kernel: within noise make test 1797/0, kernel 154/0, extensions 123/0. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(stlib.initialise-datatypes) ran the rule compiler for its two datatypes on every start: 300M of a 600M warm boot. The cold boot now records the phase with the fasl recorder and stores the stream in the sidecar, SHENSIG2. A boot with SHEN_FASL=off stores none and runs the phase, and boot_cache_spec's fingerprint still equals an uncached boot in every cache configuration. Instructions retired, millions, warm, 3 runs each: startup ~590 -> ~490 (stage delta: datatypes 300 -> 10; footprint 21 -> 20.6 MB) Specs 1921/0, kernel 154/0, extensions 123/0. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Every boot parsed types.kl (which has no declares on 43.1) and init.kl with R.read_all to fill the native signature table: 65M of 480M instructions, for a table only the native typechecker reads. The files are still read at install, as the working directory can change, and parsed at the first typecheck. A signature declared or destroyed before then stands; typecheck_lazy_spec pins it. Instructions retired, millions, warm, 3 runs each: startup ~490 -> ~410 footprint 20.5 -> 20.4 MB Programs that typecheck pay the same, later. Specs 1927/0, kernel 154/0, extensions 123/0. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Finishing a cycle sooner leaves less floating garbage in the estimate that sets the next threshold. Peak footprint, MB: kernel 186 -> 163, nrev 40-56 -> 37, extensions 33 -> 29; hof 126 -> 115-122. startup 23 -> 25, vec 25 -> 27. Instructions and CPU within noise (nrev 181.2 either way). Instructions-only numbers would show nothing here: this is a memory change. Cost: each GC step does more work at once, so individual steps are longer. SHEN_GC=400,200 restores the previous setting. Tested and not adopted, larger pauses (with stepmul 1000): 600: nrev 176 B, 12.0 s, 56 MB; hof 232 MB; kernel 263 MB 800: nrev 174 B, 11.2 s, 74 MB; hof 1133 MB; kernel 281 MB make test 1807/0, kernel 154/0, extensions 123/0. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A curried chain through a defun of arity n calls PARTIAL n-1 times. Each partial's call copied its arguments into two tables and looped over them. jit.p on the kernel suite put 8% of samples there, and more when those loops did not trace. Billions of instructions, warm, base dd24cbb -> this: kernel suite 22.3 -> 19.3 (CPU 2.6 -> 1.8 s, footprint 187 -> 170 MB, 154/0), typed load 6.1 -> 6.0; hof, fib, compile unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ists Attribution on the warm kernel suite: shen.variancy 12.2G of 15.1G, 174 calls at 70M each. It is declare's System S query on the kernel's Prolog for the function's old type, and all it can do is warn that the type changed. A replay of a name with no signature now skips it. With a signature, declare runs in full, so the warning still appears, cli_spec pins it cold, warm and with SHEN_FASL=off. A user datatype rule giving (fn F) a type for a function with no signature could have warned on the cold run and will not on a hit: that case alone differs. Instructions retired, billions, warm fasl, 2-3 runs each: kernel suite 15.1 -> 3.6 footprint 185 -> 82 MB (154/0) typed load 2.3 -> 1.4 footprint 78 -> 54 MB extensions 1.46 -> 1.39 (123/0) Specs 1932/0 outside cli_spec, cli_spec 86/0 after fixing the new test itself. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…mbdas Generated code for (F A B) with F a variable was APP(APP(F, A), B): two vararg dispatches, plus a closure from the first when F took both. Now: - APP1..4 replace APP where the count is known. - ((F a) b) is CURn(F, a, b), a direct call when F's arity is n. A last argument that is not an atom uses CURnP/PREn, which run any partial application before evaluating it, so order and errors are those of the nested form. - A symbol in the function position resolves without a partial. - A one-parameter lambda writes no FA entry. That was a new key in a weak table per closure (44% of APP's samples on hof). APP recovers its arity from debug.getinfo, memoised, on the path where the count is not 1. Lua 5.1 has no nparams and keeps MKFUN. - The helpers are vararg functions so they never anchor a root trace: as plain functions APP1 became a shared root, every loop through it linked into it, and hof's instructions ranged 11.7-20.4 B run to run. Billions of instructions, warm, previous commit -> this: hof 48.1 -> 49.9 instructions, CPU 4.4 -> 3.2 s, footprint 120 -> 54 MB (a win in time and memory, not in instructions: it removes cache and GC work); kernel suite 19.3 -> 18.1 (CPU 1.9 -> 1.56 s, footprint 170 -> 156 MB, 154/0); typed 6.0 -> 5.9; compile warm 3.32 -> 3.27, cold 25.1 -> 26.1; fib unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(lambda X (lambda Y B)), up to 4 deep, is one function: a call with every argument allocates nothing where it used to allocate a closure per step. A partial goes through PARTIAL, as for a defun. A lambda reading no outer local is built once in the defun's KC table. FNEW is a bytecode LuaJIT cannot trace, and jit.v showed every trace through fold aborting on it. Two evaluations of a closed lambda are now the same function, the only visible change. Billions of instructions, warm, previous commit -> this: hof 49.9 -> 13.5 (CPU 3.2 -> 0.71 s); kernel suite 18.1 -> 18.5 (CPU unchanged at 1.58 s, 154/0), typed 5.9 -> 6.07, within or near run-to-run noise; compile warm 3.27 -> 3.32, cold 26.1 -> 25.6; fib unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…it runs ADD, HD, APPn, setmetatable and the rest were chunk globals: a hash lookup in ENV and an identity guard per use. A never-reassigned local is an immutable upvalue, and a trace folds a function one to a constant. Only names a body mentions are bound, and wide mode binds none, as it budgets its upvalues without them. Every non-exhaustive define also read F["shen.f-error"] on entry. Billions of instructions, warm, previous commit -> this: fib 5.0 -> 4.3; hof 13.5 -> 10.2; typed 6.07 -> 5.9; kernel suite 18.5 -> 18.0 (CPU 1.58 -> 1.68 s, within noise; 154/0); compile warm 3.32 -> 3.38, cold 25.6 -> 23.9. Startup 0.75 -> 0.80. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Peak footprint was pause 400 on top of live data, not retained data: the kernel suite peaked at 186 MB with 36 MB live after a full collection. SHEN_GC=200 on dd24cbb gave 97 MB at the same instructions, but cost nrev dearly, so a fixed pause is the wrong knob. A finaliser that re-arms each cycle now sets the pause to reach live + 48 MB, clamped to 150..400, so a small heap keeps 400. An explicit SHEN_GC stays fixed, SHEN_GC=off still leaves the host's settings. Instructions retired, billions, and peak footprint, 2 runs each (before -> after): kernel suite, cold 61.5, 322 MB -> 54.4-67, 150 MB (154/0; instructions noisy) kernel suite, warm 3.6, 82 MB -> 3.6, 78 MB 1500 defines, cold 5.4, 116 MB -> 5.3, 93 MB closures (hof) 47.8, 127 MB -> 48-50, 70 MB vec, fib, str, prolog, typed: unchanged 3M-cell live list plus churn: 13.8, 1276 MB -> 17.7, 567 MB (the cost: +28%) Specs 1941/0, kernel 154/0, extensions 123/0, examples/hello_embed.lua exits 0. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…l call; calls spec A counting defun wrote SD[1] on entry and on every return, including on the clauses that make no call, half of all calls in a tree recursion. A defun's leading tests, built from primitives that change nothing, now run uncounted, and each branch opens the count if it needs it. A segment restarts the function from the top, which only repeats those tests. A redefined primitive with side effects in such a test would run twice at a segment boundary. Loop-lowered and TRMC defuns are unchanged. Also: no uncurrying in wide mode, which budgets each closure's upvalues (fixes datatype_consume_spec's 64-premise rule, broken since 5d6b380). test/calls_spec.lua covers every application shape, redefinition with arity change, evaluation order through CURnP, proper tail calls through each helper, deep recursion across segments and host functions. It passes on dd24cbb as well (36/1 there before its expectation for a let-bound (fn f) was corrected to what dd24cbb does). Billions of instructions, warm, previous commit -> this: fib 4.3 -> 3.65; hof 10.2 -> 10.1; typed 5.9 -> 5.85; kernel suite 18.0 -> 18.0 (154/0); compile warm 3.38 -> 3.25, cold 23.9 -> 25.0 (three runs of 24.9-25.1; dd24cbb 24.9-26.3 on the same load). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A Prolog query deeper than 96 counted frames opened a fresh coroutine every 96 frames, and each one regrew a stack from nothing. S.run now keeps up to 32 finished segments; the collector still shrinks an idle one's stack. Warm, 5 runs, private SHEN_KERNEL_CACHE per tree, instructions retired (G): prolog before 20.56 20.64 20.79 22.36 22.75 cpu 1.8-2.1 s peak 48 MB prolog after 15.86 15.91 15.91 16.05 17.22 cpu 0.9-1.1 s peak 40 MB Specs green with 17 checks added here; kernel 154/0; extensions 123/123. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
lazyderef and unify_core held loops that were inner loops of every trace through a predicate. LuaJIT aborts on those, so traces were cut there or fell back to the interpreter. One-step dereferences and unifications binding a single variable now run with no loop; chains and two lists go out of line. Warm, 5 runs, private SHEN_KERNEL_CACHE per tree, instructions retired (G): prolog before 15.86 15.91 15.91 16.05 17.22 cpu 0.9-1.1 s prolog after 14.25 14.72 14.92 15.10 16.28 cpu 0.8-1.05 s Specs 1814/0; kernel 154/0; extensions 123/123. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A store into a table with a metatable makes the trace guard the type of the old value in the slot. Vectors carry Vmt, which has no metamethods, so overwriting a vector with values of another type left the trace at every element. rawset is recorded with no such guard. Only the message for a non-table first argument changes (rawset's "table expected" instead of "attempt to index"). Warm, private SHEN_KERNEL_CACHE per tree, 6 runs, instructions retired (G): vec before 13.01 13.03 13.06 13.06 13.06 13.09 cpu 0.46-0.57 s vec after 6.49 6.50 6.52 6.52 6.53 6.54 cpu 0.26-0.36 s kernel suite unchanged within noise (see final A/B: 22.3-22.5 vs 22.4-22.7) Specs 1814/0; kernel 154/0; extensions 123/123. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Under dual-number LuaJIT (arm64) a self-tail loop's trace is specialised to the types at the moment it is recorded. An accumulator that starts as an int and leaves int32 range then fails the trace's guard on every iteration, running the loop's head plus a side trace, about 10 times the cost of the loop. A self-tail argument (+ V E), (+ E V), (- V E) or (* V E), for parameter V and non-literal E, now always yields a double, so the trace is specialised to that. Values and errors are the same. Constant steps stay ints, so counters and indices are untouched. x86-64 LuaJIT has no int representation to leave. Warm, private SHEN_KERNEL_CACHE per tree, instructions retired (G): vec before 6.49 6.50 6.52 6.52 6.53 6.54 cpu 0.26-0.36 s vec after 1.80 1.80 1.80 1.83 cpu 0.11-0.13 s fib 4.94-5.01 vs 4.94-5.03, loop 2.17-2.24 vs 2.16-2.21, typed 6.21 vs 6.26 Specs green with accumulator_spec added; kernel 154/0; extensions 123/123. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The kernel's shen.mod builds a list of doubled divisors and subtracts them, a few thousand instructions per call. Non-negative integers below 2^53 now take N % D, which is what that computes; everything else keeps the kernel's path. This mostly helps programs calling shen.mod directly, such as the str workload, since hash already had its own native path. It is not a string change. Warm, private SHEN_KERNEL_CACHE per tree, instructions retired (G): str before 3.77 3.80 4.03 cpu 0.28-0.35 s peak 40 MB str after 1.36 1.42 1.43 1.47 cpu 0.10-0.14 s peak 25-27 MB The string operations alone (no shen.mod or str, same counts) were already 1.36 G. Specs green; kernel 154/0; extensions 123/123. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A cons has a [1] too, and a number there read as a length: (address-> (cons 5 []) 3 8) grew the cell instead of raising. The check is now on the metatable, which also removes the type test on the length. vec, private caches, 4 runs each: 1.89 G (1.89-1.96) before, 1.88 G (1.85-1.98) after. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(if 1 a b) took the then branch with no error, since Lua treats anything but nil and false as true; shen-go, shen-cl, Shen/Scheme and Shen/JIT raise. The test of if and cond and both operands of and/or are now checked where the compiler emits them. A form that can only give a boolean goes unchecked, and anything else is BC(form). The last operand of and/or is in tail position: the check goes to the leaves of its control forms, and a call to the function being compiled stays a tail call, so a loop written (and (p X) (loop Xs)) keeps its frame. Its final value reaches whoever tests the result. A tail call to any other function there is no longer a tail call. (not 1) is unchanged here: it compiles to Lua's not and gives false. Replaces the divergence pinned in error_robustness_spec. Costs: INTEGRATION.md. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
08c6adf kept this only for a call to the function being compiled, so mutual recursion through and/or grew the stack: (ev? 5000000) with ev? and od? calling each other as (and (number? N) (od? (- N 1))) raised at 10000 stack segments, where 912596b, Shen/JIT and Shen/Scheme give true. In a defun's tail a call there is now left unchecked and its value is the function's result, which a test of it checks. In a control form in value position or a top-level form nothing is a tail call, so a call there is still checked. Stated gap: a caller that uses (and A (g X))'s value without testing it can see a non-boolean from g. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…th 5M The whole body of a freeze or a lambda compiled as a control form in value position, so a tail call there was still checked and grew the stack. It is in tail position, so it stays a tail call too. error_robustness_spec now covers mutual tail recursion at depth 5M through and, or, and if, cond, let, do, and, or, freeze and lambda inside them, and through a variable; on 32dbc61 it fails. It also pins the cases left unchecked, on purpose: Shen/JIT and Shen/Scheme raise on those, shen-go gives the value. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
pyrex41
left a comment
There was a problem hiding this comment.
Summary
The boot, fasl, boolean-test, cons-fact, count-sinking, and closed-lambda work is the right shape, and most of the conformance histories in the PR body are actually forbidden by the code. Four fast paths are wrong. A fifth finding from the first pass, that unify_core overflows by skipping seg_depth when the terms are identical, does not hold: return thawH(cont) is a tail call, so that frame does not accumulate. The old counter existed because the thaw was not in tail position.
Panel: Linus Torvalds (do not merge yet), Joel Armstrong (failure isolation on the segment pool and on PROVEN), Kyle Kingsbury (one live violated history, the symbol-head CURnP re-lookup; the other claimed properties hold in the code, several of them untested).
Issue counts by severity
- bugs: 4
- suggestions: 1
- nits: 0
| .. "end, function(g, " .. plist .. ", r, an, ...)\n" | ||
| .. " if r == D then\n" | ||
| .. " if type(g) == 'function' then return g(" .. plist .. ", an) end\n" | ||
| .. " return F[g.name](" .. plist .. ", an)\n" |
There was a problem hiding this comment.
[bug] Exact-arity CURnP looks a symbol up again after the last argument has run. PREn (prims.lua:2999) already resolved F[g.name] and then threw that function away, returning the direct-call sentinel. CURnP then does F[g.name](a1..an). Nested APP, the path taken when the arity is not n, closes the partial over the function object before an is evaluated.
(let F add2 (F 1 (do (define add2 X Y -> 0) 2))) must yield 3. This path yields 0. calls_spec only covers order-free arguments, so it does not lock the case. The PR text names this re-lookup; it is still the wrong function.
Suggestion: Have PREn return the function it just resolved, and call that object. Do not read F[g.name] again after an.
| if form == true or form == false then return true end | ||
| if not is_cons(form) then return false end | ||
| local h = car(form) | ||
| return is_symbol(h) and not env[h.name] and BOOLEAN_FORMS[h.name] == true |
There was a problem hiding this comment.
[bug] gives_boolean is "the head symbol is in this set". It does not look at arity. (if (= 1) a b) and (if (cons?) a b) compile to a partial function with no BC wrap. A function is truthy in Lua, so the test takes the then-branch and never raises. Saturated (= a b) is fine, and and/or check their own operands. This is the check the patch exists to add, skipped on the calls that do not return a boolean.
Suggestion: Skip BC only when the call has the primitive's real arity (and, for and/or/not, the shape you actually inline). Otherwise wrap it.
| r = false | ||
| if getinfo then | ||
| local i = getinfo(f, "uS") | ||
| if i.nparams == 1 and not i.isvararg and i.what ~= "main" then |
There was a problem hiding this comment.
[bug] Dropping MKFUN for one-parameter lambdas is fine. Recovering them with debug.getinfo is not. Any one-parameter, non-vararg Lua function whose chunk name does not start with @ or = is treated as a Shen lambda of arity 1: zero arguments become a partial, extra arguments are applied to the result. That includes load / loadstring of host code on LuaJIT and PUC Lua 5.2+, which this tree runs. Nothing at creation time records the fact, and RAW_LAMBDA then freezes the guess. On PUC 5.1 nparams is absent, so the guess stays false and the same source behaves differently per host.
Suggestion: Mark the closure when you build it (a weak table set from lam1, or keep MKFUN). Do not sniff source names.
| local pool, npool = {}, 0 | ||
| local POOL_MAX = 32 | ||
| local function seg_body(f, ...) | ||
| return seg_body(co_yield(DONE, pcall(f, ...))) |
There was a problem hiding this comment.
[bug] Reusing a finished coroutine is the right cost model. Two things about this wrapper are wrong.
pcall around the body keeps the coroutine alive across an error, and PUC Lua 5.1 cannot yield across pcall. LuaJIT and Lua 5.2+ can. A host yield inside a deep segment on 5.1 becomes attempt to yield across C-call boundary, which this pcall turns into a failed job, and seg_done then pools the coroutine as if the job finished (runtime.lua:368). The old runner resumed f directly and forwarded yields.
Pooling also keeps coroutine-local state. A debug.sethook installed on coroutine.running() inside a segment is still installed on the next S.run, and the one after that. This tree does not call sethook; an embedder that does now shares that hook across jobs. A fresh coroutine did not.
Suggestion: On 5.1, do not pcall the body; allocate a fresh coroutine after an error. Before pooling anywhere, drop the hook and any other coroutine-local state. Pooling the success path is enough.
| for k in pairs(facts) do | ||
| if not PROVEN[k] then PROVEN[k] = true; added[#added + 1] = k end | ||
| end | ||
| local r = f(...) |
There was a problem hiding this comment.
[suggestion] with_cons_facts records keys in the process-global PROVEN and clears them only after f returns. atomic_compile does not snapshot PROVEN. An error while compiling the branch leaves the key set. The key is a gen() name and is not issued again, so a later defun does not miscompile. The table still grows by one entry per failed cons-guarded compile, and it is the one mutated local outside the restore list.
Suggestion: Clear the keys you added if f errors, or put the previous table back on the atomic_compile failure path.
With F holding a symbol and a last argument that rebinds that name, (+ 0 (F 1 (do (rebind) 2))) called the new definition; 912596b gives the old one's result. A tail call never reached CURnP: a counting defun hoists its arguments, so it compiled as APP1(APP1(F, 1), arg) and settled the function at the first application. Non-tail calls compile as CUR2P(F, 1, PRE2(F, 1), arg), where CURnP looked the name up again after the argument. PREn now hands over the function it found. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
gives_boolean looked only at the head, so (if (cons?) a b) and at KL level (if (= 1) a b) skipped the check and took the then branch on the closure. shen-go and Shen/Scheme raise. The check is skipped now only for a saturated call. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
is_raw_lambda took any chunk with a bare name as compiled, and a host's bare loadstring has one. So APP on a host's unary function with 2 arguments errored (912596b ignores the extra), with 0 returned a partial, and a curried one with 2 curried. load_chunk now records each chunk's source name and only those count. A lookup per chunk at load and on APP's other-than-one-argument path: nothing per closure, unlike a mark at creation. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
seg_done packed fixed slots on a host's yield, so a yield of one value reached the host with trailing nils. Exactly the values yielded now pass through. On PUC Lua a hook set during a job stayed on the pooled coroutine and the next job inherited it, so a coroutine with a hook is not reused. seg_pool_spec covers both and runs on LuaJIT and PUC 5.1; the yield case runs on LuaJIT, since PUC 5.1 cannot yield across a segment's pcall on 912596b either. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
with_cons_facts undoes its keys on return, which an error skips. gen()'s counter is monotonic, so a stale key cannot match later code: this is growth, not a miscompile. A failed compile now clears them, with the rest of its state. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(not 5) compiled to Lua's not and gave false, while the primitive raises. Shen/JIT and shen-go raise, Shen/Scheme gives false. An argument that can only give a boolean is unchecked, as for the other tests. Costs: INTEGRATION.md. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
run-kernel-tests.lua exited 0 whatever the suite did: its only os.exit(1) was for a harness that would not load. A failed test, a suite stopping midway and a truncated suite all left make certify and CI green. Its Counters line read the harness's globals after kerneltests.shen's final (reset), so it always said 0. It now counts each test as it ends and exits 1 unless the suite finished, none failed and exactly 154 ran; a kernel that adds tests needs that number bumped. A real run with one test broken gives make rc 2, and 912596b's real count is 154 passed, 0 failed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Nothing in CI noticed the native typechecker falling back to the kernel's, since a fallback only prints to stderr and the suite still passes: certify now runs with SHEN_TYPECHECK_NATIVE=require. bench/golden_typecheck.lua could not run from a checkout; it now uses tests/ and also checks the y-combinator's 431741 inferences, as make golden, in CI. scripts/run-tests.lua has a fixed list, and test/system_s_sigf_spec.lua was not on it: it is now, and a spec missing from the list fails the run. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Count sinking compiles a branch, and when a test turns out to make a counted call, compiles it again. The first attempt's CCACHE entries named locals declared only in the discarded text, so the second read an undeclared global: nil, with no error. (pick 0 [0 1] 0) gave b, and an hd of a non-cons returned instead of raising. It takes all of: a lambda capturing a parameter, which leaves the defun without a self-tail loop, the only case that sinks at its top; a test with a control form in value position whose tail is a call outside LEAF, such as not, which makes the helper's call counted; and a branch that hoists an accessor chain to a local. compile_attempt now rolls back CCACHE, the local budget, KC bodies, the counted-call flags and counters, the hoist list and F references. try_trap_opt uses it as well: it compiles its default before it knows the fast path applies. No repro for that site. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
leave_as_tail_call tested for a defun, so a top-level lambda's last operand of and/or was checked, and hit the stack limit: (set top (/. N (or (= N 0) ((value top) (- N 1))))) at 3M. 912596b, Shen/JIT, Shen/Scheme and shen-go give true. The whole body of a lambda or a freeze now counts as a function's tail wherever it sits. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ments
Uncurrying changed what a host could do with a function it got back:
shen.eval("(/. X Y (+ X Y))")(1)(2) raised, and shen.eval("(/. X (/. Y (cons X Y)))")(1)
returned a malformed cons with no error. Compiled functions take exactly their arity,
and a host does not know it. shen.eval, shen.call, shen.fn, shen.totable and the interop
marshalers now give a wrapper through APP, and take it back as the function it stands
for, so internal calls cost nothing. A host's own function crosses as itself.
Stated limits, in the README and CHANGELOG: a function inside a list or vector from
shen.eval is the raw one, applied with the new shen.apply; shen.prims.F[name] is raw too.
The wrapper is a different Lua value from the raw function: the API hands out only one of
them, but Shen code holding the raw one and a host comparing with the wrapper would differ.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The recorder skipped a macro-time (set ...) when the global already held that value at compile time, so a session replaying the fasl without it saw no value at all, where an uncached run has it. Compiled with (set cfg 5) already done, then replayed fresh: unset. It now records the value once per fasl, and skips only repeats. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A kernel with no entry fails and names the table to update, rather than passing or failing on a stale 154. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
codegen_diff_spec runs 495 shapes built to reach count sinking's second compile and a fixed fuzz program (scripts/codegen-fuzz.py, from the independent review) with each optimisation switch off, and compares results. It also scans the bytecode of every chunk compiled for the kernel, stdlib and both programs for a global that is not in the compiled environment: a local whose declaration was lost reads as nil, with no error. With compile_attempt's child cache removed, which is the original bug, it fails on both counts. About 13 s. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Lua 5.5 removed collectgarbage's setpause and setstepmul, so require failed at boot, on 912596b too, though the README claims 5.5. Three deep-recursion checks in calls_spec now run on LuaJIT only: PUC raises C stack overflow on those at 912596b as well. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
It left a function raw on every route it did not cover: one fetched with shen.value, one Shen passes to a host callback, and one a host puts inside a list compared with =, which failed. The fix belongs in the function itself: next commit. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
An uncurried lambda called raw with too few arguments ran its body with nil, so a host calling a function value it got from anywhere (shen.eval, shen.value, a callback, a list) got a wrong result or an error, and so did (thaw (/. X Y ...)). Each compiled lambda now tests its last parameter on entry: nil is never a Shen value, so nil there means arguments are missing, and it returns the partial application over the ones it got (UPART). Compiled calls always pass every argument or go through APP. Checked that nil is never a Shen value in practice by logging every firing of the guard: 0 across 37 specs, the kernel suite, extensions, 9 example programs, THORN steam and the contract file; 18, all deliberate, in embed_values_spec. A host function that returns nil into Shen, passed as a lambda's last argument, would now get a partial. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…nil edge Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Combines five measured improvement branches (conform, boot, lists, calls, prims) on kernel 43.2, plus the fixes that integration and two review rounds needed. Head: 5a0228d. Every number below names the SHA it was measured on.
Performance against 912596b and Shen/JIT
Instructions retired, billions (G), Apple Silicon under load (15-40), median (range) of n interleaved runs, private kernel and fasl caches. Warm means a fasl hit; first means an empty fasl dir with a warm kernel cache, which is the fair comparison with Shen/JIT on compile-heavy rows, since Shen/JIT has no cache. Measured on 67a79f3; the two commits after it (the nil check and the changelog) measured within noise of it, see below. Ratios are head/Shen-JIT.
Behaviour changes
ifandcond, both operands ofand/or, and the argument ofnot. Before,(if 1 a b)gavea. A predicate with too few arguments is not a boolean.and/orstays a tail call and is not checked itself, so deep mutual recursion through it keeps working (tested at depth 5M). A test of the result is what raises. Compared with other ports, g returning 5:(zf 5)(zf2 5)(zt1 5)(zt2 5)(zn1 5)(zn2 5)(zn3 5)(zn4 5)(zvp 5)(zit 5)(zlm 5)(zfr 5)(zvr (/. X X) 5)(zpr1 5)(zca 5)(and true 5)(or false 5)(and true (zg 5))(if (zg 5) a b)(and (zg 5) true)(or (zg 5) true)(cond ((zg 5) a) (true b))(not (zg 5))(let X (zf 5) (+ X 1))Narrower than shen-go, wider than Shen/JIT and Shen/Scheme.
(str "a")returns the quoted form (4adf578, a separate commit you can drop), matching the other ports.absvectorlengths, out-of-rangeaddress->/<-address, non-symbolset/value,hashwith a non-positive bound, unknownget-timeoptions.(get-time unix)is Unix time;(eval-kl sym)gives the symbol;closeis idempotent; a fasl replays a macro'ssetof an unchanged value.shen.prims.Fthat returns nil, whose result is passed as a lambda's last argument, gets a partial.(thaw (/. X Y ...))gives a partial again, and thaw of a one-parameter lambda gives the lambda itself (912596b ran its body with nil). From the host only, raw over-applicationadd(1, 2, 3)gives 3, where 912596b returned a function. The nil edge can't be reached through the Lua bridge, which turns nil into(); the reviewer tried.=can be true).Bugs found and fixed since 32dbc61
and/orwere lost (strict-test regression): in defuns, freeze and lambda bodies, and top-level lambdas.set.make certifyexited 0 whatever the suite did, and its Counters line always read 0. Now it fails on a failure, an early stop, or a count other than the kernel's (154 for 43.2, kept per kernel version). Shown: one test broken gives make rc 2.Verification on 5a0228d
Code SHA cc2a162; head 5a0228d differs from it only in CHANGELOG.md, and the specs were re-run on 5a0228d (2373/0).
make certifyandmake goldenrun the way CI runs them: rc 0,Counters: passed=154 failed=0 expected=154, golden 27/27.SHEN_TYPECHECK_NATIVE=requireas well.benchmark-coveragepasses on both engines. hello_embed works, and the single-file bundle boots from another directory.Known and not in this PR
(shen.mod 5 0)hangs on all five ports tested.(shen.mod 7 -3)hangs here, on Shen/Scheme and on shen-go; Shen/JIT raises.extensions/andbenchmarks/from shen-sources, which aren't vendored; CI can't run them. Here they were run against Shen/JIT's 43.2 copies.🤖 Generated with Claude Code