Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 17 additions & 2 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,18 +7,33 @@ Notable changes to shen-lua. Release notes for earlier versions are under

### Changed

- **Kernel: ShenOSKernel 43.4** (community `Shen-Language/shen-sources`, tag
`shen-43.4`, commit `f34b307b490880eedcfe1f5e5d3cc24ca3d2602e`, release
tarball SHA-256 `15e7972e...a2756`). Vendored byte-identical; `(version)` is
`"43.4"`. `tests/` is the 43.4 suite. The native typechecker's fingerprints
are unchanged (verified with `scripts/kernel-fingerprints.lua`).
- stlib.kl now declares its 290 signatures with
`shen.declare-with-signature` and a precompiled abstraction, so the
`<kernel cache>.stlib` signature sidecar is gone. `typecheck_native.lua`
wraps `shen.declare-with-signature`; without that its native signature
table silently lacked the stdlib signatures (171 instead of 465 entries).
The two stdlib datatypes still run the rule compiler; their fasl records
stay in `<kernel cache>.stlibdt`.
- **Startup and compile cost.** `variable?` no longer throws on a non-symbol; the
fasl records a global push as a delta (SHENFASL9), the reader has native paths for
plain tokens, macroexpansion keeps unchanged cells, the stdlib datatypes replay from
the signature sidecar (SHENSIG2), and a replayed `declare` skips the variancy query
`<kernel cache>.stlibdt`, and a replayed `declare` skips the variancy query
when the compile run found no old type. The GC pause adapts past a small live heap
and stops adapting once the host sets its own; the default stepmul is 1000, which
trades work for memory (a string-building loop: +7% instructions, 78 to 47 MB).
- **Calls and closures.** Fixed-arity `APPn`/`CURn` helpers, uncurried lambda groups,
closed lambdas built once, hot helpers bound as locals, count sinking. `(= F G)` can
now be true for two evaluations of the same closed lambda.
- **Lists, vectors, strings, Prolog.** `hd`/`tl` read directly where `(cons? E)`
proves a cons; `address->` stores with `rawset`; a self-tail accumulator is a double
proves a cons, and a `let` bound to a cons keeps that fact. `+`/`-`/`*` and
the comparisons are Lua operators once both sides are numbers; `/` stays
checked, because Lua division by zero does not raise. `address->` stores with
`rawset`; a self-tail accumulator is a double
from the start; native `shen.mod` below 2^53; the Prolog engine reuses stack
segments and has loop-free deref and unify.
- **Kernel cache key** covers every codegen switch: `SHEN_UNCURRY`,
Expand Down
6 changes: 3 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
# shen-lua

[Shen](https://shenlanguage.org) on **LuaJIT 2.1**: pattern matching, optional
sequent types, and Prolog, compiled to Lua. ShenOSKernel 43.2, **154/154**
sequent types, and Prolog, compiled to Lua. ShenOSKernel 43.4, **154/154**
official kernel tests. Embeds in any Lua host. Plain Lua 5.1/5.4/5.5 works too (slower,
still correct).

Expand Down Expand Up @@ -147,7 +147,7 @@ More under [`examples/`](examples/README.md). A full tour:

```sh
make test # port specs (test/*_spec.lua)
luajit run-kernel-tests.lua # official 43.2 suite → 154/154
luajit run-kernel-tests.lua # official 43.4 suite → 154/154
```

The kernel suite is vendored in `tests/`. Port specs cover primitives, REPL,
Expand All @@ -160,7 +160,7 @@ luarocks install shen # launcher + modules
luarocks make --local shen-scm-1.rockspec # this tree
```

LuaJIT required (`lua == 5.1`). This tree uses ShenOSKernel **43.2**; release
LuaJIT required (`lua == 5.1`). This tree uses ShenOSKernel **43.4**; release
**0.12.0** is the first on the community 43.2 kernel, **0.11.1** uses Tarver's S42 kernel; **0.9.0** was 41.1. Or grab `shen-bundle.lua` from
[Releases](https://github.com/pyrex41/shen-lua/releases/latest) — one file,
`require("shen-bundle")`.
Expand Down
4 changes: 2 additions & 2 deletions bin/yggdrasil-build.lua
Original file line number Diff line number Diff line change
Expand Up @@ -108,8 +108,8 @@ end
assert(man.kernel, MANIFEST .. ": missing kernel=")
assert(man.init, MANIFEST .. ": missing init=")
assert(#man.user > 0, MANIFEST .. ": no user= entries")
if man["kernel-version"] ~= "43.2" then
io.stderr:write(("yggdrasil-build: warning: manifest kernel-version=%s, this port is certified against 43.2\n")
if man["kernel-version"] ~= "43.4" then
io.stderr:write(("yggdrasil-build: warning: manifest kernel-version=%s, this port is certified against 43.4\n")
:format(tostring(man["kernel-version"])))
end

Expand Down
167 changes: 32 additions & 135 deletions boot.lua
Original file line number Diff line number Diff line change
Expand Up @@ -183,7 +183,7 @@ local function find_kldir()
end
local KLDIR = find_kldir() .. "/"
P.KLDIR = KLDIR -- resolved .kl directory (trailing /), for typecheck_native
-- Boot order for the ShenOSKernel-43.2 kernel (community shen-sources line).
-- Boot order for the ShenOSKernel-43.4 kernel (community shen-sources line).
-- Every 43.1 .kl file is pure defuns: no module has load-time side effects, so
-- the order below (upstream make.shen's module order, then the booted
-- extensions, then the precompiled standard library) is for readability only.
Expand Down Expand Up @@ -223,7 +223,7 @@ P.GLOBALS["*implementation*"] = rawget(_G, "jit") and "LuaJIT" or _VERSION
P.GLOBALS["*port*"] = "shen-lua"
P.GLOBALS["*porters*"] = "shen-lua contributors"
P.GLOBALS["*os*"] = (package.config and package.config:sub(1,1) == "\\") and "Windows" or "Unix"
P.GLOBALS["*release*"] = "0.1" -- port release; kernel *version* comes from init.kl ("43.2")
P.GLOBALS["*release*"] = "0.1" -- port release; kernel *version* comes from init.kl ("43.4")

-- ---- kernel bytecode cache -------------------------------------------------
-- Loading the kernel from .kl sources costs ~0.8s (read + parse + KL->Lua
Expand Down Expand Up @@ -537,8 +537,9 @@ end
-- (ShenOSKernel 43.1 has no top-level forms at all — its kernel signatures are
-- precompiled closures in init.kl's shen.initialise-signedfuncs — so on the
-- vendored kernel hoist_tail finds nothing to hoist. The machinery stays for
-- S42-shaped trees via SHEN_KL_DIR, and record_declares / replay_declares are
-- reused for the standard library's 290 declares; see stlib_types.)
-- S42-shaped trees via SHEN_KL_DIR. Since 43.3 the stdlib's 290 signatures are
-- precompiled in stlib.kl and are not replayed here; the datatype rule
-- compiles are cached separately (stlib_datatypes).)
-- On the S42.0 kernel, klambda/types.kl ended with 161 top-level
-- `(declare Name Type)` forms, and
-- they are not cheap annotations: `declare` (types.kl) runs the type theory for
Expand Down Expand Up @@ -1517,46 +1518,26 @@ end
-- defuns stay loaded but are not registered as Shen functions).
local STDLIB_LOADED = false

-- (stlib.initialise-types) is 290 (declare Name Type) calls, and `declare`
-- runs the type theory for real on each (variancy under the Prolog machine,
-- then a KL->Lua compile of the signature's prolog abstraction): ~0.2 s, the
-- bulk of a warm boot. Everything it depends on is fixed by the kernel cache
-- key, so cache it exactly the way the S42 kernel's own types.kl signatures
-- were cached (hoist_tail / record_declares / replay_declares): a sidecar file
-- next to the kernel bytecode cache holds, per signature, its raw type and the
-- dumped abstraction chunk, plus the gensym / inference counters. A warm boot
-- replays them into shen.*sigf* (and the native typechecker's signature
-- table) in the original order. SHEN_KERNEL_CACHE=off disables it with the
-- kernel cache.
-- ShenOSKernel 43.3 precompiles the 290 stdlib signature abstractions as
-- (shen.declare-with-signature Name Type (freeze Abstraction)) inside
-- stlib.kl. A warm boot no longer runs the type theory per signature, and the
-- old <kernel cache>.stlib signature sidecar is gone. typecheck_native wraps
-- shen.declare-with-signature, so its signature table still sees every one.
--
-- The file also holds the record stream of (stlib.initialise-datatypes), in the
-- user fasl's format: its two datatypes run the whole rule compiler and were
-- 40% of a warm boot's instructions. The records need the fasl recorder, so a
-- boot with SHEN_FASL=off stores none and runs the phase.
local SIGS_FORMAT = "SHENSIG2" -- 2: datatype records after the counters
local function stlib_sigs_path()
if P.KERNEL_CACHE_DATA then return nil end -- single-file bundle
-- The two datatypes still run the rule compiler (shen.process-datatype), which
-- was the expensive half of a warm boot once the signatures were precompiled.
-- Their fasl records live in <kernel cache>.stlibdt (SHENDT1), keyed on the
-- kernel cache key. SHEN_FASL=off, SHEN_KERNEL_CACHE=off, or a bundle with no
-- cache path runs the phase for real. A truncated file is a miss and is
-- rewritten; a replay that raises still raises.
local DT_FORMAT = "SHENDT1"
local function stlib_dt_path()
if P.KERNEL_CACHE_DATA then return nil end
local p = cache_path()
return p and (p .. ".stlib") or nil
return p and (p .. ".stlibdt") or nil
end

local function write_sigs(path, key, decl, counters, datatypes)
local parts = { SIGS_FORMAT, "\n", key, "\n", tostring(#decl), "\n" }
for _, e in ipairs(decl) do
parts[#parts+1] = e.name .. "\n" .. #e.dump .. "\n" .. e.dump
kdata_ser(e.typ, parts)
end
parts[#parts+1] = tostring(counters.gensym) .. "\n" .. tostring(counters.infs) .. "\n"
parts[#parts+1] = datatypes or ""
local tmp = path .. ".tmp"
local fh = open_for_write(tmp)
if not fh then return end
fh:write(table.concat(parts)); fh:close()
os.remove(path)
os.rename(tmp, path)
end

local function read_sigs(path, key)
local function read_datatypes(path, key)
local data = read_file(path)
if not data then return nil end
local pos = 1
Expand All @@ -1566,54 +1547,17 @@ local function read_sigs(path, key)
local s = data:sub(pos, e - 1); pos = e + 1
return s
end
if line() ~= SIGS_FORMAT or line() ~= key then return nil end
local n = tonumber(line() or ""); if not n then return nil end
local decl = {}
local ok = pcall(function()
for i = 1, n do
local nm = line()
local len = tonumber(line() or "")
if not nm or not len or pos + len - 1 > #data then error("truncated") end
local dump = data:sub(pos, pos + len - 1)
pos = pos + len
local typ
typ, pos = kdata_de(data, pos)
decl[i] = { name = nm, typ = typ, dump = dump }
end
end)
if not ok then return nil end
local gensym = tonumber(line() or ""); if not gensym then return nil end
local infs = tonumber(line() or ""); if not infs then return nil end
local datatypes
if pos <= #data then
datatypes = fasl_parse(data, pos)
if not datatypes then return nil end
end
return { decl = decl, gensym = gensym, infs = infs, datatypes = datatypes }
end

-- The sidecar, read once per boot: both phases below use it.
local STLIB_SIGS, STLIB_SIGS_READ
local function stlib_cached()
if not STLIB_SIGS_READ then
STLIB_SIGS_READ = true
local path = stlib_sigs_path()
if path then
STLIB_SIGS = read_sigs(path, kernel_key() .. "|" .. SIGS_FORMAT)
end
end
return STLIB_SIGS
if line() ~= DT_FORMAT or line() ~= key then return nil end
return fasl_parse(data, pos)
end

-- (stlib.initialise-datatypes), from its records when the sidecar has them,
-- else for real, recording them for write_sigs (stlib_types, the next phase
-- that matters) when the fasl recorder is live.
local STLIB_DT_BLOB
local function stlib_datatypes()
local run = P.F["stlib.initialise-datatypes"]
local cached = stlib_cached()
if cached and cached.datatypes then return fasl_replay(cached.datatypes) end
if cached or not FASL_INSTALLED or not stlib_sigs_path() then return run() end
local path = stlib_dt_path()
if not path or not FASL_INSTALLED then return run() end
local key = kernel_key() .. "|" .. DT_FORMAT
local cached = read_datatypes(path, key)
if cached then return fasl_replay(cached) end
local rec = { n = 0, in_chunk = false }
local arity0 = {}
for k, v in pairs(C.ARITY) do arity0[k] = v end
Expand All @@ -1627,59 +1571,12 @@ local function stlib_datatypes()
C.NO_KDATA = saved_nokdata
if not ok then error(err, 0) end
local sok, blob = pcall(fasl_serialize, rec, arity0)
if sok then STLIB_DT_BLOB = blob end
if sok then atomic_write(path, DT_FORMAT .. "\n" .. key .. "\n" .. blob) end
end

local function stlib_types(verbose)
local init_types = P.F["stlib.initialise-types"]
local path = stlib_sigs_path()
if not path then return init_types() end
local key = kernel_key() .. "|" .. SIGS_FORMAT
local tn = package.loaded["typecheck_native"]
local on_sig = tn and tn.record_sig or nil
local cached = stlib_cached()
if cached then
-- load every dump before running any, so a foreign-arch file falls back
-- to the real declares with nothing half-applied
local fns, lok = {}, true
for i, e in ipairs(cached.decl) do
local ok, fn = pcall(P.load_chunk, e.dump, "declare:" .. e.name)
if not ok then lok = false; break end
fns[i] = fn
end
if lok then
replay_declares(cached.decl, fns, on_sig)
if type(P.GLOBALS["shen.*gensym*"]) == "number"
and cached.gensym > P.GLOBALS["shen.*gensym*"] then
P.GLOBALS["shen.*gensym*"] = cached.gensym
end
if type(P.GLOBALS["shen.*infs*"]) == "number"
and cached.infs > P.GLOBALS["shen.*infs*"] then
P.GLOBALS["shen.*infs*"] = cached.infs
end
if verbose then io.stderr:write(" stdlib signatures (cached)\n") end
return
end
os.remove(path)
end
-- Cold: collect the (name, type) pairs the compiled initialiser declares,
-- then run the real declares through record_declares in the same order.
local live = P.F["declare"]
local decls = {}
P.F["declare"] = function(name, typ)
decls[#decls + 1] = { name = name, typ = typ }
return name
end
local ok, err = pcall(init_types)
P.F["declare"] = live
if not ok then error(err, 0) end
local out = record_declares(decls)
if out then
write_sigs(path, key, out, {
gensym = type(P.GLOBALS["shen.*gensym*"]) == "number" and P.GLOBALS["shen.*gensym*"] or 0,
infs = type(P.GLOBALS["shen.*infs*"]) == "number" and P.GLOBALS["shen.*infs*"] or 0,
}, STLIB_DT_BLOB)
end
if verbose then io.stderr:write(" stdlib signatures\n") end
P.F["stlib.initialise-types"]()
end

local STDLIB_PHASES = { "environment", "arities", "macros", "synonyms",
Expand All @@ -1695,7 +1592,7 @@ local function load_stdlib(verbose)
local hush0 = P.GLOBALS["*hush*"]
local ok, err = pcall(function()
-- (stlib.initialise) is exactly these phases in this order; run them
-- individually so the types phase can be served from the cache.
-- individually so the datatype phase can be served from the cache.
local whole = true
for _, ph in ipairs(STDLIB_PHASES) do
if not P.F["stlib.initialise-" .. ph] then whole = false end
Expand Down
Loading
Loading