Repository navigation
skdb: pass INT64 fields to JS as raw i64 (BigInt) instead of text - #1722
Merged
Merged
Conversation
The OJS marshalling path sent every out-of-int32-range integer as text: n.toString() in wasm, then importString + parseInt in the TS client. Two problems: the formatting/parsing cost per value, and silent precision loss for |n| > 2^53 (parseInt returns a double). Give the wasm extern an Int (i64) parameter instead, so the integer crosses as a BigInt with no string work. On the JS side, keep the field a number when the value is within the safe-integer range (as before, so e.g. millisecond timestamps keep their type) and use bigint beyond 2^53-1, where number was silently wrong. The native stub in runtime64_specific.cpp takes int64_t; the OJS path is a no-op natively as before. Measured with a node benchmark against the wasm dev build (select of 8192 INT rows with out-of-int32 values materialized as JS objects, median of 50 runs, interleaved A/B): 15.1-15.4 ms before vs 2.1-2.3 ms after, i.e. ~7x faster (~1.6 us saved per int64 row). Generated by Mistral Vibe. Co-Authored-By: Mistral Vibe <vibe@mistral.ai> Co-Authored-By: Claude Opus 5.5 (Anthropic, via Mistral Vibe; thinking: high) <noreply@anthropic.com>
Generated by Mistral Vibe. Co-Authored-By: Mistral Vibe <vibe@mistral.ai> Co-Authored-By: Claude Opus 5.5 (Anthropic, via Mistral Vibe; thinking: high) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
The OJS marshalling path (wasm skdb -> JS objects) sent every
out-of-int32-range integer as text:
n.toString()in wasm, thenimportString+parseIntin the TS client(
sql/ts/src/skdb_skdb.ts). That had two costs: formatting/parsingper value, and silent precision loss for |n| > 2^53, since
parseIntreturns a double (the old path returned e.g.9007199254740992for9007199254740993).This gives the wasm extern an
Int(i64) parameter, so the integercrosses as a
BigIntwith no string work. On the JS side the fieldstays a
numberwithin the safe-integer range (so millisecondtimestamps and all previously-exact values keep their type) and
becomes an exact
bigintbeyond 2^53-1.Changes:
sql/src/SqlValue.sk:jsPushObjectFieldInt64(Int), call sitepasses
ndirectly.sql/ts/src/skdb_skdb.ts: import is(v: bigint) => void; thehandler assigns
Number(v)when safe, elsev(ToWasm interface,LinksImpl, Manager wiring).
skiplang/prelude/runtime/runtime64_specific.cpp: native stubtakes
int64_t(OJS is a no-op natively, as before).sql0.3.1 -> 0.3.2,std0.8.2 -> 0.8.3.API note: fields holding Int64 values with |n| > 2^53-1 change type
from
number(lossy) tobigint(exact). Values within the saferange are unchanged.
Measurement
Node benchmark against the wasm dev build (
createSkdb({asWorker: false}), 8192 INT rows with out-of-int32-range values,select i from tmaterialized as JS objects, median of 50 runs,interleaved A/B with both variants built from the same tree):
~7x faster end-to-end on int64-heavy results (~1.6 us saved per row).
Validation
tsctypecheck, eslint, prettier, skfmt clean on touched files.42,2^31-1,2^31,1700000000001,2^53-1come back as exactnumber;2^53+1, max int64(
9223372036854775807), and min+1 come back as exactbigint.(Big values are computed arithmetically in SQL because int
literals beyond 2^53 don't lex exactly on wasm — a pre-existing
String.toIntlimitation, unrelated to this change.)output unchanged (
42|2147483648|1700000000001|9007199254740993).sql/unit_tests.shwith the native binary: 83 OK, 0 FAILED.Follow-up (not in this PR)
The JSON wire protocol (client<->server, tail) still sends Int64 as
unquoted JSON numbers (
SqlCAst.sk:223), whichJSON.parsereads asdouble — same >2^53 precision loss over the wire. Fixing that means
quoting large ints (or a bigint-aware parse) plus protocol version
negotiation.
Notes
trailers); validation performed in-session as described above.