Repository navigation
Avoid decimal strings for safe Arrow decimal values - #633
Conversation
20681dc to
e09c427
Compare
| return `${negative ? '-' : ''}${digits.slice(0, cut)}.${digits.slice(cut)}`; | ||
| } | ||
|
|
||
| function decimalToNumber(value: Uint32Array & Parameters<typeof bigNumToBigInt>[0]): number { |
There was a problem hiding this comment.
nit: Could this get a short doc comment? For example: "Decimal128 is four little-endian 32-bit words. If words 2–3 are pure sign extension of word 1, the unscaled value fits in an int64, and we return it directly when it's a safe integer; otherwise fall back to bigNumToBigInt."
aakash-saravanan-db
left a comment
There was a problem hiding this comment.
LGTM. The sign-extension checks and safe-unscaled-integer cutoff preserve the existing rounding, and sharing the helper across scalar and nested conversion keeps the optimization at the right boundary. Independent output comparisons matched the base, and local benchmarks reproduced the conversion speedup.
Non-blocking follow-up, reproduced on both the base and this PR: with preserveBigNumericPrecision: true, unscaled -9007199254740993 at scale 2 returns "-90071992547409.92" instead of "-90071992547409.93". Arrow 13's bigNumToBigInt returns a string for Decimal128 despite its declared type, so unary negation in bigNumDecimalToString coerces it through a JS number and loses precision. Please track a separate fix to normalize to an actual bigint before formatting, with regression coverage for large negative decimals in scalar and nested results. This is preexisting and does not block this optimization.
|
@bot add a short doc comment to decimalToNumber. Example: "Decimal128 is four little-endian 32-bit words. If words 2–3 are pure sign extension of word 1, the unscaled value fits in an int64, and we return it directly when it's a safe integer; otherwise fall back to bigNumToBigInt." |
|
Pushed |
Signed-off-by: peco-engineer-bot[bot] <peco-engineer-bot[bot]@users.noreply.github.com>
|
Thanks for your contribution! To satisfy the DCO policy in our contributing guide every commit message must include a sign-off message. One or more of your commits is missing this message. You can reword previous commit messages with an interactive rebase ( |
Signed-off-by: Vu Anh Phung <vu.phung@databricks.com> Co-authored-by: Isaac <no-reply@databricks.com>
Signed-off-by: Vu Anh Phung <vu.phung@databricks.com> Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com> Signed-off-by: Vu Anh Phung <vu.phung@databricks.com>
Assert the stubbed vector.get feeds getRows, and add unsafe values that must take the BigInt fallback, so the byteLength checks cannot pass vacuously if getRows stops reading through the stubs. Co-authored-by: Isaac <no-reply@databricks.com> Signed-off-by: Vu Anh Phung <vu.phung@databricks.com>
Signed-off-by: peco-engineer-bot[bot] <peco-engineer-bot[bot]@users.noreply.github.com>
5dab295 to
b9ea5cf
Compare
Follows #628 (merged); rebased onto
main.Arrow Decimal128 → number currently formats an intermediate decimal string for every value. Profiles attribute 60% of decoded mixed-batch time and 6.67 seconds of a live kernel lineitem 2M run to this conversion.
Read the unscaled integer directly from its existing 32-bit words after checking both upper words are valid sign extensions. Conservatively restrict this fast path to safe integers; extending it to all signed 64-bit values is a follow-up, not a requirement for identical rounding. Larger values and precision-preserving output retain the original conversion. Both backends share the helper, including nested/legacy rows; BIGINT, IPC, defaults, logging and telemetry are unchanged.
Verification: 1,837 unit tests pass with inherited agent markers cleared (the default-environment failure also reproduces on main), all static checks pass, and 32 real live e2e cases plus type/limit/lifecycle checksums match main/#628. Mutants disabling the fast path, excluding negatives, or dropping upper-word guards fail tests. Decoded mixed-object materialization falls 71.8% (4,844 → 1,365 ms).
Live Node 24.22.1 medians below use five rotated, fresh-process pairs, timing execute + materializing drain. These compare against #628, not pre-#628 main.
Lineitem/object p95/p99: Thrift 22.09 → 13.48 s; kernel 19.42 → 11.07 s (with five runs, both percentiles are maxima). Decoded unsafe/wide/precise median changes range from −3.2% to +0.7%; live string/small controls show A/A noise, including 16-row Thrift +5.5%. Mixed-object peak RSS medians are 623 → 631 MiB; no small-query or blanket memory win is claimed.
Adversarial review: converged in round 1; gains and parity independently reproduced, signed fast-path mutants now fail, conservative cutoff retained and inherited precision issue deferred.
This pull request and its description were written by Isaac.