Repository navigation
Conversation
pyrex41
left a comment
There was a problem hiding this comment.
Summary
Draft spike. Both flags stay off unless the environment value is exactly on, and the kernel cache key includes them. I would not merge it. Let-insertion of a non-escaping unary lambda is a reasonable experiment, and the tests are a real oracle for the histories they name (shared state, escaping per-item closures, redefinition, evaluation order, nested maps, improper-list fallback), because flag-off is the ordinary evaluator. The pass is wrong on a closure that mentions itself, on a callback that binds let or lambda, and on a compile error in the specialized callback. The map worker is a second copy of map. The bench dumps do not belong in git. The writeup's own numbers say the closure pass eliminates 2 of 737 kernel defuns, which does not earn a permanent compiler flag.
Panel: Linus Torvalds (no), Joel Armstrong (compile error in the callback poisons the next compile), Kyle Kingsbury (self-reference is the history the specs do not forbid).
Issue counts by severity
- bugs: 4
- suggestions: 1
- nits: 0
| if not R.is_cons(x) then return x end | ||
| local a=array(x) | ||
| if name(a[1])==target then | ||
| return list{R.intern('let'),lam[2],replace(a[2],target,lam),lam[3]} |
There was a problem hiding this comment.
[bug] Eliminated calls are rewritten to (let Param Arg Body) and then compiled as ordinary impl code (IN_IMPL true, SELF live). A real lambda is compiled with IN_IMPL false and SELF nil, so known-arity calls inside it read F["name"] at application time and a self-call is not a tail-loop continue. After elimination, those calls use the entry-time fN = F["name"] local (or impl) and a tail (rec …) can be lowered to goto tco. Redefining a callee after the outer function is entered but before the inlined call, including from the argument expression, is ignored. Immediate applications at line 110 have the same hole.
Suggestion: Compile the planted body on the nested-lambda path (call-time F[], no self-tail lowering), or refuse elimination when the body contains a known-arity call or an outer self-call.
| elseif R.is_cons(v) then for _,w in ipairs(array(v)) do captures(w) end end | ||
| end | ||
| captures(lam[3]); for n in pairs(seen) do r.captures[#r.captures+1]=n end | ||
| table.sort(r.captures); uses(a[4],r.binding,false,r) |
There was a problem hiding this comment.
[bug] uses walks the let body and never the lambda. (let F (lambda X (if (= X 0) 0 (F (- X 1)))) (F 3)) is one unsaturated call and zero escapes, so it is eliminated. replace drops the binding and pastes the body, which still calls F, at the call site. F is then unbound. A mutual pair is the same hole: the inner binding looks unused, is deleted, and the outer body still names it. test/closure_ir_spec.lua never puts the binding in its own body.
Suggestion: Walk the lambda body in uses. A reference to the binding there is an escape. Do not inline it.
| end | ||
| count(lam[3]); if nodes > 80 then return end | ||
| local fv, caps = {}, {} | ||
| collect_free(args[1], env, {}, fv) |
There was a problem hiding this comment.
[bug] The worker is a chunk-scope KC function, so a free variable that collect_free misses is not an upvalue of impl. It is a nil global. collect_free treats let and lambda as syntax when env[head] is unset, and it does not look at bound. A callback that binds a variable named lambda or let still has the body parsed as that special form.
(map (lambda X (let lambda X (lambda Y Y))) L) with outer parameter Y is a call of the list item on the outer Y. Y is classified as the false lambda's binder, omitted from the capture list, and the worker runs with Y nil. Normal MKFUN nesting would have closed over Y. The worker is also a second copy of native map (cons tag, reverse, MAP_TAIL), gated on function identity. One consumer, one hard-coded representation.
Suggestion: Treat let/lambda as syntax only when that name is not in bound, matching cexpr. Otherwise refuse the specialization. Do not grow a private map per higher-order function.
| local saved_self, saved_h, saved_cc, saved_fnl = SELF, HOIST, CCACHE, FNL | ||
| local saved_impl, saved_cont = IN_IMPL, push_inline_cont() | ||
| SELF=nil; HOIST=nil; CCACHE={}; FNL=new_fnl(true); IN_IMPL=false | ||
| local body = cexpr(lam[3], extend(env, lam[2].name, item)) |
There was a problem hiding this comment.
[bug] try_map_closure installs fresh SELF, HOIST, CCACHE, FNL, IN_IMPL, and CONT, then calls cexpr on the lambda body. The restores on the next lines run only if cexpr returns. There is no pcall. With the flag on, (defun mc (L) (map (lambda X (trap-error 1)) L)) errors in catom and leaves CTX set. The next top-level (if true 1 2) compiles to KC[1]() in a chunk that declares no KC. Default off does not protect the enabled path. cdefun on this base has the same success-only restore; this function adds another way into it, and its own saves are not on the failure path.
Suggestion: pcall the callback compile and restore those locals before rethrowing. Give cdefun the same finally for CTX, CONT, SELF, IN_IMPL, DEFUN_NAME, FREF_*, FNL, CCACHE, and C.ARITY[name].
| @@ -0,0 +1,10 @@ | |||
| LuaJIT 2.1.1774638290 arm64 jit=true iterations=1000000 | |||
There was a problem hiding this comment.
[suggestion] bench/results/ is one machine's timings, LuaJIT -jv logs, JSON dumps, and datatype-dispatch.patch (a source diff, not a measurement). The spike writeup already quotes the medians that match closure-experiments.txt. The trace logs will rot on the next machine and make every rerun a diff.
Suggestion: Keep the harnesses and the short summary the markdown cites. Drop bench/results/.
|
Closing as a recorded negative result, not merging. The spike's own write-up says the closure pass eliminates 2 of 737 kernel defuns, which doesn't justify a permanent compiler flag. The write-up stays available at SPIKE-CLOSURES.md, and the branch is kept. Review findings, checked by running them with the flags off and on at 37a646b:
|
Closure construction interrupts tracing in compiler tree traversals and higher-order code. This spike adds opt-in local lambda elimination (
SHEN_CLOSURES=on) and guarded native-map specialization (SHEN_MAP_CLOSURES=on), passing captures directly while materializing ordinary closures for replacement consumers. Both flags remain off by default.Fresh-process comparisons show 27.5% faster uncached interpreter loading and 20.3% faster stateful-object execution; retained memory is unchanged and generated kernel Lua grows 5.1%. Reports, benchmark drivers, and raw evidence are in
SPIKE-CLOSURES.mdandSPIKE-CALLBACKS.md.Typechecking measurements remain variable even after flushing boot traces.
SPIKE-TYPECHECK.mdisolates query work and profiles native datatype dispatch. Unrolled predicate dispatch was slower and remains benchmark-only. Explicit loop dispatch won five of six repeat pairs (72 ms versus 125 ms median), but overlapping ranges, a shared host, and one workload do not establish a general speedup. The typechecker runtime is unchanged; both prototypes and semantic comparisons are retained underbench/for further investigation.Validation: 1,128 port checks and 134 canonical kernel tests pass in both default and combined-enabled configurations. Focused closure cases cover shared state, escaping closures, redefinition, evaluation order, nested maps, and malformed-list/error fallback. Datatype experiments additionally pass 57 unrolled-dispatch and 54 loop-dispatch checks for inference counts, order, redefinition, cuts, errors, missing predicates, and positive/negative typing. Python harness syntax and Lua file loading were checked. The archived temporary datatype patch applies cleanly but is not active runtime code.