Skip to content

prelude: vendor Ryu, make String.toFloat correctly rounded (fix WASM round-trip + TOML float parsing) - #1287

Merged
mbouaziz merged 15 commits into
SkipLabs:mainfrom
mbouaziz:ryu-string-to-float
Nov 13, 2026
Merged

mbouaziz merged 15 commits into
SkipLabs:mainfrom
mbouaziz:ryu-string-to-float

Conversation

@mbouaziz

@mbouaziz mbouaziz commented Jul 7, 2026 •

Copy link
Copy Markdown
Contributor

What

Two pre-existing float-parsing bugs share one root cause — reconstructing a
double from its decimal digits with repeated * 0.1, which is not correctly
rounded:

  1. WASM String.toFloat (SKIP_String__toFloat_raw's in-tree atof on
    SKIP32): "0.3".toFloat() yielded 0.30000000000000004, round-trip
    (toString().toFloat() == value) failed for ~85% of sampled doubles, and the
    leading - was ignored, so every negative number parsed to 0
    ("-1.5" → 0).
  2. TOML parseFloat: "3.14" parsed to 3.1400000000000006 instead of the
    nearest double.

Both were masked while Float.toString produced only 9 significant digits; they
surface once toString becomes exact (#1288).

How

  • Vendor the Ryu library (ryu/d2s.c double→string and ryu/s2d.c
    string→double) from ulfjack/ryu @ 4c0618b0, with #ifdef SKIP32 guards so it
    builds under -nostdlibinc, and RYU_ONLY_64_BIT_OPS on SKIP32 so it needs no
    128-bit compiler-rt helpers (__multi3, __lshrti3). ryu/README.md records
    the origin, license and local changes. (d2s is used by prelude: Float.toString via Ryu (round-trip-safe, shortest representation) #1288; s2d here.)
  • SKIP32 String.toFloat uses Ryu s2d (correctly rounded, ~20 ns) for the
    inputs it supports (≤17 significant digits). For longer inputs it calls a new
    host import, SKIP_js_parse_float, implemented in sk_runtime.ts with
    Number() (correctly rounded for any input). The in-tree atof is removed.
    SKIP64 keeps libc atof, already correct.
  • TOML parseFloat keeps its validation but delegates the value to
    String.toFloat instead of reconstructing it by hand.

Wasm import: the module now imports SKIP_js_parse_float. Every host
(skdb, @skipruntime/wasm) gets it from sk_runtime.ts, which their builds
copy in next to the module, so they stay in sync. A host that loads a new
module with an old sk_runtime.ts would fail to instantiate.

Tests

  • runtime/tests/string_to_float.{c,mjs}: s2d built with the Wasm runtime's
    flags, checked bit for bit against Number() (edge cases, 100k random decimal
    strings, shortest representations of 100k random doubles). Links without
    --allow-undefined, so it fails if __multi3 & co. come back.
  • runtime/tests/to_float_raw.{c,mjs}: the real string.c built for Wasm, with
    the host import as in sk_runtime.ts; checks the s2d path and the host
    path against Number(). Fails on the previous atof
    (-123456789012345678 → 0).
  • toml float-bits: compares decoded bits, which the JSON fixtures cannot
    (they compare 9-digit toString output). Fails on the previous parseFloat.

All three run in CI: the first two in the compiler job's runtime regressions
(<1s), the last in the toml package tests.

Choice of Ryu, license

  • Compatibility with JS: Ryu d2s gives the same shortest digits as JS
    Number.prototype.toString (0 differences over 7M doubles, including
    subnormals and DBL_MAX). Differences in layout (1.0 vs 1, exponent
    thresholds) belong to the prelude: Float.toString via Ryu (round-trip-safe, shortest representation) #1288 formatter.
  • License: Ryu is dual Apache 2.0 / Boost 1.0; we use it under Boost, which
    is MIT-compatible and only requires keeping the notices in source copies.

Validation

Rebased onto main (one conflict in prelude/build.sk: magic.c is now
Wasm-only since the build-id change; kept that and added the Ryu sources and
.include("runtime")). main's incremental prelude build (#1637) tracks
runtime/*.c/*.h only, so runtime/ryu/ is added to RUNTIME_SRCS
(checked: touching ryu/s2d.c rebuilds the archive).

  • Vendored files compared with upstream @ 4c0618b0: d2s and licenses
    byte-identical; s2d.c/ryu_parse.h differ only by the clang-format pass.
  • s2d vs libc strtod: 0 mismatches over ~25M random inputs, with both the
    native 128-bit path and the RYU_ONLY_64_BIT_OPS path used on Wasm (UBSan
    trap mode).
  • skdb npm package built locally and run in Node (real sk_runtime.ts): long
    FLOAT literals (-123456789012345678, 3.14159265358979323846, …) insert
    as the nearest double. skdb Wasm module: the only new import is
    SKIP_js_parse_float; no __multi3/__lshrti3.
  • @skip-wasm/std builds and lints.
  • runtime/tests/run.sh, prelude (131/131) and toml (332/332) tests pass.
  • Not run locally: make STAGE=1, the skdb Wasm Playwright suite (CI runs both
    with the candidate toolchain since the runtime changed).

After merge, the CI images need a docker-publish run (runtime change; see
HACKING.md).

AI assistance

The original commits were written with Claude Code (Claude Opus 4.6/4.8). The
rebase, review and the commits on top (tests, .gitignore, the Wasm fallback
fix, ryu/README.md) were done with Mistral Vibe (Claude Opus 5.5, thinking:
high); see the Co-Authored-By trailers.

@jberdine jberdine left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Not sure if the CI job itself failed or there is something off.

@mbouaziz

Copy link
Copy Markdown
Contributor Author

I guess I just need to restart the CI (rebase needed anyways)
See #1335 for why

@mbouaziz
mbouaziz force-pushed the ryu-string-to-float branch 2 times, most recently from 9f691c5 to bb93897 Compare November 13, 2026 10:42
@mbouaziz
mbouaziz enabled auto-merge November 13, 2026 10:51
mbouaziz and others added 15 commits November 13, 2026 10:52
Vendored unmodified from https://github.com/ulfjack/ryu at commit
4c0618b0e44f7ef027ebae05d2cc7812048f7c8f (2026-02-09).

Files copied from ryu/ directory:
  ryu.h, d2s.c, common.h, d2s_full_table.h, d2s_intrinsics.h, digit_table.h

License files copied from repository root:
  LICENSE-Apache2, LICENSE-Boost

Ryu converts IEEE 754 doubles to their shortest decimal string
representation. It is self-contained C, dual-licensed under
Apache License 2.0 and Boost Software License 1.0.

No modifications applied — files are byte-identical to upstream.

## How to reproduce

  COMMIT=4c0618b0e44f7ef027ebae05d2cc7812048f7c8f
  DST=skiplang/prelude/runtime/ryu
  mkdir -p "$DST"
  for f in ryu.h d2s.c common.h d2s_full_table.h d2s_intrinsics.h digit_table.h; do
    curl -sL "https://raw.githubusercontent.com/ulfjack/ryu/$COMMIT/ryu/$f" -o "$DST/$f"
  done
  for f in LICENSE-Apache2 LICENSE-Boost; do
    curl -sL "https://raw.githubusercontent.com/ulfjack/ryu/$COMMIT/$f" -o "$DST/$f"
  done

## How to verify

Compare vendored files against upstream checksums (sha256):

  common.h         0bbd71d26da6193e678d0776cf418f43f287c73d6fd6725353df0aadf70f2a19
  d2s.c            d24323c7eb77d63f1e52c415212b50060b776f0883525d907d04954fcd48cf64
  d2s_full_table.h 2618f6e5fae6c4443899b184efe3d08295dd267dc9f1a994c983c7caca59ebe6
  d2s_intrinsics.h 1d05702f2edacce428223d4b43dd3095c1bd1f3ad30128ce1761d84356dddc7d
  digit_table.h    8b782573abc0b8554d74163ae6c02f0beb5c30d1e63eaebdb0ddf2c98d817e01
  ryu.h            b7feab0ba1df5e9ef3d602f386592ad152491405e49b408fb21c0e0e9e6bfb16
  LICENSE-Apache2  c71d239df91726fc519c6eb72d318ec65820627232b2f796219e87dcf35d0ab4
  LICENSE-Boost    c9bff75738922193e67fa726fa225535870d2aa1059f91452c411736284ad566

Run: sha256sum skiplang/prelude/runtime/ryu/*

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add Ryu-specific clang-format settings to minimize diff against
upstream. Based on LLVM style with no column limit, cast spacing,
and short-if-on-single-line to match Ryu's conventions.

Add clang-format off/on guards around lookup tables to preserve
upstream's aligned two-entries-per-line layout.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Guard libc headers behind #ifdef SKIP32 since WASM is compiled with
-nostdlibinc (no libc available):

- d2s.c: replace <assert.h>, <stdlib.h>, <string.h> with SKIP32 stubs
  (memcpy declaration + assert no-op); guard d2s() which uses malloc
- common.h: replace <assert.h>, <string.h> with SKIP32 stubs
- d2s_intrinsics.h: guard <assert.h> with #ifndef SKIP32
- ryu.h: guard <inttypes.h> with #ifndef SKIP32

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add ryu/d2s.c to both the Makefile and build.sk so it's compiled for
SKIP64 and SKIP32 targets, and list runtime/ryu/ in the prelude
Makefile's RUNTIME_SRCS so that editing a Ryu file rebuilds the runtime
archive.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Vendored from https://github.com/ulfjack/ryu at commit
4c0618b0e44f7ef027ebae05d2cc7812048f7c8f (the same commit as the d2s
files), then reformatted with the vendored ryu/.clang-format.

Files copied from the ryu/ directory:
  s2d.c, ryu_parse.h

Ryu's s2d converts a decimal string to the nearest IEEE 754 double
(correctly rounded). Upstream documents it as experimental: it supports
up to 17 significant digits and not all input formats, returning a
non-SUCCESS Status otherwise (callers fall back for those cases). s2d
reuses the shared d2s tables/intrinsics already vendored; no new tables.

Upstream sha256 (pre-format):
  s2d.c        9485e78f329ec1cf931b737333ef7bd85195864dfb9dd096413f0d336a7a46d3
  ryu_parse.h  da8ad9764b2701627cb75926b9d14b540fdfd7be693d671da9594eb16dabd27a

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Same treatment as d2s.c: guard libc headers behind #ifdef SKIP32 since
WASM is compiled with -nostdlibinc (no libc):

- replace <assert.h>, <stdlib.h>, <string.h> with SKIP32 stubs
  (memcpy declaration + assert no-op)
- guard the s2d() convenience wrapper, which uses strlen; callers on
  SKIP32 use s2d_n() with an explicit length instead

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Compile ryu/s2d.c for both SKIP64 and SKIP32 (added to runtime/Makefile
CFILES and prelude/build.sk srcs, alongside ryu/d2s.c).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
On SKIP32 (WASM) there is no libc strtod, so SKIP_String__toFloat_raw
fell back to the in-tree atof, which reconstructs the value with repeated
`* 0.1` and is not correctly rounded. This breaks round-trip: e.g.
"0.3".toFloat() yielded 0.30000000000000004, and toString().toFloat()
did not recover the original for most values.

Use Ryu's s2d (correctly rounded) on SKIP32 for the inputs it supports
(<=17 significant digits, standard formats), falling back to the existing
atof for the rest (>17 digits, leading '+', etc.). SKIP64 keeps using
libc atof, which is already correctly rounded.

Verified: s2d round-trips all shortest-representation toString outputs
(0 failures over 7053 sampled doubles, incl. subnormals and DBL_MAX),
where the old atof failed ~85%.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
TOML's parseFloat reconstructed the value by hand with repeated `* 0.1`,
which is not correctly rounded: e.g. "3.14" parsed to 3.1400000000000006
rather than the nearest double. The old 9-digit Float.toString masked
this, but it breaks round-tripping and shows up once toString becomes
exact (shortest-representation).

Keep the existing validation walk (so malformed input is still rejected
and the TOML "invalid" cases still fail), but delegate the actual value
to String.toFloat (Ryu s2d on WASM, libc strtod on native), after
stripping digit-group underscores and a leading '+'.

All 331 toml tests pass.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
wasm32 defines __SIZEOF_INT128__, so Ryu's `HAS_UINT128` path used
__uint128_t. Those 128-bit ops lower to compiler-rt calls (__multi3,
__lshrti3) that are not available under -nostdlibinc, so the resulting
wasm module failed to instantiate:

  LinkError: WebAssembly.instantiate(): Import "env" "__multi3":
  function import requires a callable

Define Ryu's `RYU_ONLY_64_BIT_OPS` under `#ifdef SKIP32` in
d2s_intrinsics.h to force the portable 64-bit path (used by both d2s and
s2d). Verified: no 128-bit libcalls remain in either wasm object, and
s2d still round-trips all 7053 sampled toString outputs. SKIP64 is
unaffected.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The runtime Makefile now builds ryu/d2s.c and ryu/s2d.c into ryu/*.o and
ryu/*.bc, which the existing runtime/*.o and runtime/*.bc patterns do not
cover (*.o is also ignored at the root, *.bc is not). Ignore both.

Generated by Mistral Vibe.
Co-Authored-By: Mistral Vibe <vibe@mistral.ai>
Co-Authored-By: Claude Opus 5.5 (Anthropic, via Mistral Vibe; thinking: high) <noreply@anthropic.com>
Build ryu/s2d.c the way the Wasm runtime does (SKIP32, -nostdlibinc) and
check it bit for bit against JavaScript's correctly-rounded Number(): fixed
edge cases (ties to even, subnormals, DBL_MAX, overflow, underflow), 100k
random decimal strings and the shortest representations of 100k random
doubles. Linking without --allow-undefined also catches 128-bit compiler-rt
calls (__multi3, __lshrti3): with RYU_ONLY_64_BIT_OPS removed from
d2s_intrinsics.h the link fails.

Runs in the compiler job's "Run runtime regressions" step (~0.3s).

Generated by Mistral Vibe.
Co-Authored-By: Mistral Vibe <vibe@mistral.ai>
Co-Authored-By: Claude Opus 5.5 (Anthropic, via Mistral Vibe; thinking: high) <noreply@anthropic.com>
The JSON fixtures compare Float.toString output, which has at most 9
significant digits, so they could not see the 1-ulp errors of the old
hand-rolled parser ("3.14" -> 3.1400000000000006). Compare the decoded
bits instead. Fails on the previous parseFloat, passes with String.toFloat.

Generated by Mistral Vibe.
Co-Authored-By: Mistral Vibe <vibe@mistral.ai>
Co-Authored-By: Claude Opus 5.5 (Anthropic, via Mistral Vibe; thinking: high) <noreply@anthropic.com>
On SKIP32, String.toFloat inputs that Ryu's s2d rejects (more than 17
significant digits) fell back to an in-tree atof that:

- ignored a leading '-': "-123456789012345678" parsed to 0 (before s2d,
  every negative number did, e.g. "-1.5" -> 0);
- was a few ulps off: "3.14159265358979323846" -> 3.1415926535897967
  instead of 3.141592653589793.

Replace it with a new host import, SKIP_js_parse_float, implemented in
sk_runtime.ts with Number(), which is correctly rounded for any input.
s2d still handles the common case in Wasm (~20 ns vs ~100 ns per call
into JS). Native builds keep libc atof.

The new runtime regression test builds the real string.c for Wasm and
checks both paths against Number(); it fails on the previous atof. Also
checked end to end with the skdb npm package in Node: long FLOAT literals
now insert as the nearest double.

Generated by Mistral Vibe.
Co-Authored-By: Mistral Vibe <vibe@mistral.ai>
Co-Authored-By: Claude Opus 5.5 (Anthropic, via Mistral Vibe; thinking: high) <noreply@anthropic.com>
Record the upstream commit, which files were taken, the SKIP32 patches
on top, and how to update. Ryu is dual-licensed Apache 2.0 / Boost 1.0;
state that we use it under the Boost license, which is MIT-compatible and
only requires keeping the notices in source copies.

Generated by Mistral Vibe.
Co-Authored-By: Mistral Vibe <vibe@mistral.ai>
Co-Authored-By: Claude Opus 5.5 (Anthropic, via Mistral Vibe; thinking: high) <noreply@anthropic.com>
@mbouaziz
mbouaziz force-pushed the ryu-string-to-float branch from bb93897 to d3a5cae Compare November 13, 2026 10:53
@mbouaziz
mbouaziz merged commit 5baa561 into SkipLabs:main Nov 13, 2026
23 checks passed
@mbouaziz
mbouaziz deleted the ryu-string-to-float branch November 13, 2026 11:15
mbouaziz added a commit that referenced this pull request Nov 16, 2026
…tion) (#1288)

## What

Replace the pure-Skip `Float.toString`, which printed at most 9
significant
digits and so could not round-trip, with a C helper built on Ryu `d2s`
(vendored
in #1287). It produces the shortest decimal string that round-trips
through the
IEEE-754 double: `toString().toFloat() == value` for every finite value,
e.g.
`0.1` prints `"0.1"` and `0.1 + 0.2` prints `"0.30000000000000004"`.

The layout is unchanged: fixed notation for 1e-4 <= value < 1e15, `.0`
for
integers, exponents `e+N`, `e-0N` and `e-NN`. Only the number of digits
changes,
e.g. `DBL_MAX` prints `1.7976931348623157e+308` instead of
`1.797693135e+308`.

**JS compatibility:** the digits are those of JS
`Number.prototype.toString`.
This fixes skdb's JS API, which passes `FLOAT` values to JS as
`Float.toString()` text and `parseFloat`s them, so values lost
everything past
the 9th significant digit.

## How

- `SKIP_Float_toStringShortest` (`string.c`) lays out Ryu's shortest
digits in
Skip's conventions; `floatToString` keeps handling `nan`, `inf` and the
sign.
- `skc` no longer constant-folds `Float.toString()`: folding uses the
compiler's own `Float.toString`, i.e. that of the prelude the compiler
was
built with, so a constant could print differently from the same value at
  runtime.
- Tests: `tests/Float.sk` expectations updated to full precision, and
previously non-round-tripping values (`5e-324`, smallest normal,
`DBL_MAX`,
2^53, 2^53 - 1) added to the round-trip test. The TOML golden for long
pi now
  has all its digits. A new runtime regression
(`runtime/tests/float_to_string.{c,mjs}`) builds the formatter for Wasm
and
  checks it against JS for 200k values.
- Versions: `std` 0.4.0 → 0.5.0 (visible output change), `skc` 0.4.0 →
0.4.1.

## Testing Float.toString with any compiler

`skc` folds `toString()` on a Float literal at compile time with its
*own*
`Float.toString`, i.e. that of the prelude the compiler was built with.
CI's
candidate toolchain is built from the bootstrap IR, which still has the
9-digit
formatter, so literal-based tests saw `5e-324` → `"4.940656458e-324"`
whatever
the runtime does. The `Float.toString` tests now parse their inputs with
`String.toFloat` (never folded), so they test the runtime formatter with
any
compiler; with the previous formatter they fail as expected. Literal
folding
goes away for good once the bootstrap is bumped past this PR.

`test_vtable_slots` compared function text including debug metadata
numbers
(`!dbg !N`), which are per LLVM module. With this PR's smaller prelude,
the
edited function lands in the same module as the checked one and
renumbers it,
while the vtable slot is unchanged. The test now strips `!dbg`
attachments
(with the existing `llvm_artifact_diff.without_dbg_attachments`) and
still
fails if the slot offset changes.

## Validation

Rebased onto `main` `e46671a9` (the #1287 commits dropped; one conflict
in
`runtime/tests/README.md` with the new macOS section, both kept). The
version
bumps are on top of #1683, which already moved `std` and `skc` to 0.4.0.

- Formatter fuzzed natively over 4M doubles (UBSan, stack protector): 0
round-trip failures, 0 layout differences from the old conventions,
output
at most 23 characters. The later simplification of the formatter (109 →
47
  lines) produces byte-identical output on 18.6M values.
- With a from-source stage1: `prelude` 131/131, `toml` 332/332, compiler
`Runtime`/`Native`/`Syntax` tests 378/378 (all `Float*` tests), skdb
native
  SQL suite against sqlite3 437/437, `runtime/tests/run.sh` pass.
- CI's path (candidate stage0 from the bootstrap, fresh build cache):
  `prelude` 131/131; with the previous formatter, the Float tests fail.
- Backend regressions, both shards, stage1: 30/30 modules (incl.
  `test_vtable_slots`, also passing on `main`).
- `skdb` Wasm build: +27 KB (Ryu's `d2s` tables).

## AI assistance

The original commits were written with Claude Code (Claude Opus 4.6).
The
rebase, review and the commits on top (rename, cleanup, tests, version
bumps)
were done
with Mistral Vibe (Claude Opus 5.5, thinking: high); see the
`Co-Authored-By`
trailers.
mbouaziz added a commit to SkipLabs/skjs that referenced this pull request Nov 18, 2026
Skip's String.toFloatOption is now correctly rounded and Float.toString
prints the shortest round-trip digits (Ryu, SkipLabs/skip#1287/#1288), so
normalizeJSNumber can name every literal by the String() form of its
double directly: parse (radix digits converted to decimal first), print
with jsNumberToString. This replaces the exact decimal pipeline
(normalizeDecimalJS, parseRadixInt, normalizeUnsignedJSNumber) and the
second beyond-precision pass (canonicalizeBeyondPrecision).

Long zero-padded spellings that exceed the runtime parser's input limit
are compacted first (compactDecimal), and a leading '.' gets a 0, so the
pinned edge spellings keep their names. No behavior change: the full
suite and a 2,000-literal differential run against node agree.

Generated by Mistral Vibe.
Co-Authored-By: Mistral Vibe <vibe@mistral.ai>
mbouaziz added a commit to SkipLabs/skjs that referenced this pull request Nov 18, 2026
…1073)

Now that Skip parses floats correctly rounded and prints them with Ryu
(SkipLabs/skip#1287, #1288), `normalizeJSNumber` names each literal
directly: it parses the literal to a double, then prints that double
with `jsNumberToString` (the JS `String()` form).

This removes the exact decimal pipeline (`normalizeDecimalJS`,
`parseRadixInt`, `normalizeUnsignedJSNumber`) and the second
beyond-precision pass (`canonicalizeBeyondPrecision`).
`src/JSNumericString.sk` goes from 276 changed lines to a net reduction
of about 90 lines.

Commits (tests first):
1. `test:` pins the spellings the runtime parser does not accept as
written: `.5`, `5.`, zero padding past the 255-byte parser limit, and a
literal that is still too long after the padding is dropped.
2. `refactor:` makes the switch. A leading `.` gets a `0` prepended, and
long zero-padded spellings are compacted first (`compactDecimal`), so
those pins keep their names.

There is no behavior change, so there is no version bump. Both commits
pass `make test` locally. A differential run on 2,000 random literals
(decimal, leading/trailing dot, exponent, hex, binary, separators)
matches node's `String(Number(x))` on every literal.

— Mistral Vibe
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants