Skip to content

skdb: pass INT64 fields to JS as raw i64 (BigInt) instead of text - #1722

Merged
mbouaziz merged 2 commits into
mainfrom
skdb-int64-bigint
Nov 20, 2026
Merged

mbouaziz merged 2 commits into
mainfrom
skdb-int64-bigint

Conversation

@mbouaziz

Copy link
Copy Markdown
Contributor

What

The OJS marshalling path (wasm skdb -> JS objects) sent every
out-of-int32-range integer as text: n.toString() in wasm, then
importString + parseInt in the TS client
(sql/ts/src/skdb_skdb.ts). That had two costs: formatting/parsing
per value, and silent precision loss for |n| > 2^53, since
parseInt returns a double (the old path returned e.g.
9007199254740992 for 9007199254740993).

This gives the wasm extern an Int (i64) parameter, so the integer
crosses as a BigInt with no string work. On the JS side the field
stays a number within the safe-integer range (so millisecond
timestamps and all previously-exact values keep their type) and
becomes an exact bigint beyond 2^53-1.

Changes:

  • sql/src/SqlValue.sk: jsPushObjectFieldInt64(Int), call site
    passes n directly.
  • sql/ts/src/skdb_skdb.ts: import is (v: bigint) => void; the
    handler assigns Number(v) when safe, else v (ToWasm interface,
    LinksImpl, Manager wiring).
  • skiplang/prelude/runtime/runtime64_specific.cpp: native stub
    takes int64_t (OJS is a no-op natively, as before).
  • Patch bumps: sql 0.3.1 -> 0.3.2, std 0.8.2 -> 0.8.3.

API note: fields holding Int64 values with |n| > 2^53-1 change type
from number (lossy) to bigint (exact). Values within the safe
range are unchanged.

Measurement

Node benchmark against the wasm dev build (createSkdb({asWorker: false}), 8192 INT rows with out-of-int32-range values,
select i from t materialized as JS objects, median of 50 runs,
interleaved A/B with both variants built from the same tree):

variant run 1 run 2
text + parseInt (base) 15.1 ms 15.4 ms
raw i64 + BigInt 2.1 ms 2.3 ms

~7x faster end-to-end on int64-heavy results (~1.6 us saved per row).

Validation

  • tsc typecheck, eslint, prettier, skfmt clean on touched files.
  • Correctness (wasm): 42, 2^31-1, 2^31, 1700000000001,
    2^53-1 come back as exact number; 2^53+1, max int64
    (9223372036854775807), and min+1 come back as exact bigint.
    (Big values are computed arithmetically in SQL because int
    literals beyond 2^53 don't lex exactly on wasm — a pre-existing
    String.toInt limitation, unrelated to this change.)
  • Native dev build compiles/links with the new stub; native text
    output unchanged (42|2147483648|1700000000001|9007199254740993).
  • sql/unit_tests.sh with the native binary: 83 OK, 0 FAILED.
  • CI (incl. the skdb-wasm playwright suite) on the branch head.

Follow-up (not in this PR)

The JSON wire protocol (client<->server, tail) still sends Int64 as
unquoted JSON numbers (SqlCAst.sk:223), which JSON.parse reads as
double — same >2^53 precision loss over the wire. Fixing that means
quoting large ints (or a bigint-aware parse) plus protocol version
negotiation.

Notes

  • AI assistance: commits generated with Mistral Vibe (Co-Authored-By
    trailers); validation performed in-session as described above.

mbouaziz and others added 2 commits November 19, 2026 23:48
The OJS marshalling path sent every out-of-int32-range integer as
text: n.toString() in wasm, then importString + parseInt in the TS
client. Two problems: the formatting/parsing cost per value, and
silent precision loss for |n| > 2^53 (parseInt returns a double).

Give the wasm extern an Int (i64) parameter instead, so the integer
crosses as a BigInt with no string work. On the JS side, keep the
field a number when the value is within the safe-integer range (as
before, so e.g. millisecond timestamps keep their type) and use
bigint beyond 2^53-1, where number was silently wrong.

The native stub in runtime64_specific.cpp takes int64_t; the OJS
path is a no-op natively as before.

Measured with a node benchmark against the wasm dev build (select of
8192 INT rows with out-of-int32 values materialized as JS objects,
median of 50 runs, interleaved A/B): 15.1-15.4 ms before vs 2.1-2.3
ms after, i.e. ~7x faster (~1.6 us saved per int64 row).

Generated by Mistral Vibe.
Co-Authored-By: Mistral Vibe <vibe@mistral.ai>
Co-Authored-By: Claude Opus 5.5 (Anthropic, via Mistral Vibe; thinking: high) <noreply@anthropic.com>
Generated by Mistral Vibe.
Co-Authored-By: Mistral Vibe <vibe@mistral.ai>
Co-Authored-By: Claude Opus 5.5 (Anthropic, via Mistral Vibe; thinking: high) <noreply@anthropic.com>
@mbouaziz
mbouaziz enabled auto-merge November 19, 2026 23:55
@mbouaziz
mbouaziz merged commit 9e7fc63 into main Nov 20, 2026
22 checks passed
@mbouaziz
mbouaziz deleted the skdb-int64-bigint branch November 20, 2026 00:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant