Skip to content

Materialize Arrow batches directly into result chunks - #641

Closed
vuanhphung wants to merge 3 commits into
mainfrom
perf/drain-copy
Closed

vuanhphung wants to merge 3 commits into
mainfrom
perf/drain-copy

Conversation

@vuanhphung

@vuanhphung vuanhphung commented Nov 20, 2026 •

Copy link
Copy Markdown
Collaborator

Not a shipping candidate: extended verification found materialized-object regressions, so this PR remains a draft for evidence only. Main spends 402 ms copying 80M row references in concatChunks (405 ms in a live kernel profile).

The prototype lets the shared Arrow converter append into the slicer's output and materialize overflow into an owned tail, preserving row conversion, chunk boundaries and clear-on-take behavior. It improves the placeholder drain but increases materialized-object heap/GC costs; three capacity-allocation variants did not fix that regression and are not included.

Verification: 1876 unit tests, 57 live e2e tests, 64 main/candidate checksum/lifecycle comparisons, build, type-check, lint and formatting pass. Fresh Node 24.22.1 processes; five rotated local pairs, real IPC, no network (sample p95/p99 are both the maximum):

Scenario Main median / p95 ms Prototype median / p95 ms
80M boolean placeholders 882 / 913 682 / 707
10M materialized objects 482 / 502 544 / 565
5M objects, 128 MiB heap 275 / 282 334 / 346
10M materialized arrays 1099 / 1199 1094 / 1120

Exact-dashboard and bounded materialized live comparisons cover both backends; shared-warehouse wall times are mixed/noisy, not a uniform speedup. The 5M-object CPU profiles show GC self time 31 → 50 ms; observed peak heap rises roughly 46 → 104 MiB, while forced-GC retained heap stays equal.

Adversarial review: not requested; self-verification fails the materialized-read safety bar. Do not mark ready or merge.

This pull request and its description were written by Isaac.

Signed-off-by: Vu Anh Phung <vu.phung@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
Signed-off-by: Vu Anh Phung <vu.phung@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
Signed-off-by: Vu Anh Phung <vu.phung@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
@vuanhphung

Copy link
Copy Markdown
Collaborator Author

Closing as no gain: writing batches straight into the slicer's output chunk saved 22.7% on an 80M-row drain with row materialization off, but slowed materialized object reads by 13–21% (more GC, about 2x peak heap), and three capacity variants didn't remove the regression.

@vuanhphung vuanhphung closed this Nov 20, 2026

This branch was previously deployed

1 inactive deployment
azure-prod — 2560403e Deployed Nov 20, 2026 by vuanhphung via e2e-test (24.22.1, 24_22_1) #2174
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant