Skip to content

Avoid decimal strings for safe Arrow decimal values - #633

Merged
vuanhphung merged 5 commits into
mainfrom
perf/row-convert-2
Nov 20, 2026
Merged

vuanhphung merged 5 commits into
mainfrom
perf/row-convert-2

Conversation

@vuanhphung

@vuanhphung vuanhphung commented Nov 19, 2026 •

Copy link
Copy Markdown
Collaborator

Follows #628 (merged); rebased onto main.

Arrow Decimal128 → number currently formats an intermediate decimal string for every value. Profiles attribute 60% of decoded mixed-batch time and 6.67 seconds of a live kernel lineitem 2M run to this conversion.

Read the unscaled integer directly from its existing 32-bit words after checking both upper words are valid sign extensions. Conservatively restrict this fast path to safe integers; extending it to all signed 64-bit values is a follow-up, not a requirement for identical rounding. Larger values and precision-preserving output retain the original conversion. Both backends share the helper, including nested/legacy rows; BIGINT, IPC, defaults, logging and telemetry are unchanged.

Verification: 1,837 unit tests pass with inherited agent markers cleared (the default-environment failure also reproduces on main), all static checks pass, and 32 real live e2e cases plus type/limit/lifecycle checksums match main/#628. Mutants disabling the fast path, excluding negatives, or dropping upper-word guards fail tests. Decoded mixed-object materialization falls 71.8% (4,844 → 1,365 ms).

Live Node 24.22.1 medians below use five rotated, fresh-process pairs, timing execute + materializing drain. These compare against #628, not pre-#628 main.

Query / row mode Thrift #628 → this PR Kernel #628 → this PR
lineitem 2M / object 20.25 → 12.23 s 18.21 → 10.45 s
lineitem 2M / array 20.11 → 12.14 s 17.41 → 10.41 s
decimal 2M / object 5.36 → 1.87 s 4.46 → 1.42 s
nested 200K / object 3.60 → 2.51 s 3.09 → 2.24 s

Lineitem/object p95/p99: Thrift 22.09 → 13.48 s; kernel 19.42 → 11.07 s (with five runs, both percentiles are maxima). Decoded unsafe/wide/precise median changes range from −3.2% to +0.7%; live string/small controls show A/A noise, including 16-row Thrift +5.5%. Mixed-object peak RSS medians are 623 → 631 MiB; no small-query or blanket memory win is claimed.

Adversarial review: converged in round 1; gains and parity independently reproduced, signed fast-path mutants now fail, conservative cutoff retained and inherited precision issue deferred.

This pull request and its description were written by Isaac.

@vuanhphung
vuanhphung marked this pull request as ready for review November 19, 2026 17:21

@peco-review-bot peco-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔀 Cross-reviewed: two independent reviews (Claude Opus 5.5 · GPT-6.1 Sol), checked by a judge model (Claude Sonnet 5.5).

✅ No issues identified.

@vuanhphung
vuanhphung changed the base branch from perf/row-convert to main November 19, 2026 20:07

@peco-review-bot peco-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔀 Cross-reviewed: two independent reviews (Claude Opus 5.5 · GPT-6.1 Sol), checked by a judge model (Claude Sonnet 5.5).

✅ No issues identified.

@peco-review-bot peco-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔀 Cross-reviewed: two independent reviews (Claude Opus 5.5 · GPT-6.1 Sol), checked by a judge model (Claude Sonnet 5.5).

✅ No issues identified.

return `${negative ? '-' : ''}${digits.slice(0, cut)}.${digits.slice(cut)}`;
}

function decimalToNumber(value: Uint32Array & Parameters<typeof bigNumToBigInt>[0]): number {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: Could this get a short doc comment? For example: "Decimal128 is four little-endian 32-bit words. If words 2–3 are pure sign extension of word 1, the unscaled value fits in an int64, and we return it directly when it's a safe integer; otherwise fall back to bigNumToBigInt."

@aakash-saravanan-db aakash-saravanan-db left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. The sign-extension checks and safe-unscaled-integer cutoff preserve the existing rounding, and sharing the helper across scalar and nested conversion keeps the optimization at the right boundary. Independent output comparisons matched the base, and local benchmarks reproduced the conversion speedup.

Non-blocking follow-up, reproduced on both the base and this PR: with preserveBigNumericPrecision: true, unscaled -9007199254740993 at scale 2 returns "-90071992547409.92" instead of "-90071992547409.93". Arrow 13's bigNumToBigInt returns a string for Decimal128 despite its declared type, so unary negation in bigNumDecimalToString coerces it through a JS number and loses precision. Please track a separate fix to normalize to an actual bigint before formatting, with regression coverage for large negative decimals in scalar and nested results. This is preexisting and does not block this optimization.

@vuanhphung

Copy link
Copy Markdown
Collaborator Author

@bot add a short doc comment to decimalToNumber. Example: "Decimal128 is four little-endian 32-bit words. If words 2–3 are pure sign extension of word 1, the unscaled value fits in an int64, and we return it directly when it's a safe integer; otherwise fall back to bigNumToBigInt."

@peco-engineer-bot

Copy link
Copy Markdown
Contributor

Pushed 703d675 to perf/row-convert-2. Started from 82aa4c0: decimalToNumber doc comment added (lib/result/ArrowResultConverter.ts:195); build/type-check/prettier/tests pass; lint blocked: airbnb-base config missing.

peco-engineer-bot Bot added a commit that referenced this pull request Nov 20, 2026
Signed-off-by: peco-engineer-bot[bot] <peco-engineer-bot[bot]@users.noreply.github.com>

@peco-review-bot peco-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔀 Cross-reviewed: two independent reviews (Claude Opus 5.5 · GPT-6.1 Sol), checked by a judge model (Claude Sonnet 5.5).

✅ No issues identified.

@github-actions

Copy link
Copy Markdown

Thanks for your contribution! To satisfy the DCO policy in our contributing guide every commit message must include a sign-off message. One or more of your commits is missing this message. You can reword previous commit messages with an interactive rebase (git rebase -i main).

@peco-review-bot peco-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔀 Cross-reviewed: two independent reviews (Claude Opus 5.5 · GPT-6.1 Sol), checked by a judge model (Claude Sonnet 5.5).

✅ No issues identified.

vuanhphung and others added 4 commits November 20, 2026 04:12
Signed-off-by: Vu Anh Phung <vu.phung@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
Signed-off-by: Vu Anh Phung <vu.phung@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
Signed-off-by: Vu Anh Phung <vu.phung@databricks.com>
Assert the stubbed vector.get feeds getRows, and add unsafe values that
must take the BigInt fallback, so the byteLength checks cannot pass
vacuously if getRows stops reading through the stubs.

Co-authored-by: Isaac <no-reply@databricks.com>
Signed-off-by: Vu Anh Phung <vu.phung@databricks.com>
Signed-off-by: peco-engineer-bot[bot] <peco-engineer-bot[bot]@users.noreply.github.com>

@peco-review-bot peco-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔀 Cross-reviewed: two independent reviews (Claude Opus 5.5 · GPT-6.1 Sol), checked by a judge model (Claude Sonnet 5.5).

✅ No issues identified.

@vuanhphung
vuanhphung added this pull request to the merge queue Nov 20, 2026
Merged via the queue into main with commit 7667e36 Nov 20, 2026
23 checks passed

This branch was previously deployed

1 inactive deployment
azure-prod — b9ea5cfc Deployed Nov 20, 2026 by vuanhphung via e2e-test (22.23.4, 22_23_4) #2183
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants