Skip to content

Release completed CloudFetch body-reader listeners - #634

Open
vuanhphung wants to merge 5 commits into
mainfrom
perf/result-memory
Open

vuanhphung wants to merge 5 commits into
mainfrom
perf/result-memory

Conversation

@vuanhphung

@vuanhphung vuanhphung commented Nov 19, 2026 •

Copy link
Copy Markdown
Collaborator

Completed Thrift CloudFetch bodies stay reachable from idle pooled TLS sockets. The anchor is in Node core: https.Agent#createConnection's 'session'/'close' socket listeners share a closure context with cb (captured by the proxy-tunnel cleanupAndPropagate), which pins each pooled TLS socket's first ClientRequest, then node-fetch's request 'error' closure, the Response, and the body-reader closures holding the settled body. Plain HTTP retains nothing, and later requests on the same socket aren't pinned. A live post-drain heap snapshot attributes 154.7 MB of large backing stores to that path after a 216 MB read. The workaround can go once Node no longer captures cb in those listeners.

Remove only the listeners installed by each body read once it settles, preserving earlier/later listeners and late-error handling. Rebased onto main after #627 merged. Concurrency, retries, telemetry, values and defaults are unchanged; kernel downloading is native and does not use node-fetch.

Verification: 1809 unit tests (31 CloudFetch), lint/format/types/build, live Thrift 9/9 and kernel 7/7, and value/type/null/limit/close/cancel checksums pass. Late-error coverage runs independently of cwd. An HTTPS keep-alive test on the production download path (real node-fetch, HttpRetryPolicy, keep-alive https.Agent, per-run localhost certificate shared with the HttpConnection proxy tests) asserts the downloaded 1 MiB buffer is garbage-collectable (WeakRef + gc) while its socket stays pooled; it fails with the listener cleanup removed, while the same test over plain HTTP passes with or without it. Independent review reruns against 4831873, fresh alternating processes on Node 24.22.1; medians:

Scenario (runs per side) Peak RSS MiB, #627 → branch Elapsed s, #627 → branch
HTTPS mock, 1 GiB, 30 ms delay (7) 856 → 435 (-49%) 1.902 → 1.779
Same mock, 5 concurrent (7) 2358 → 1078 (-54%) 4.017 → 3.665
Mock, 256 MiB materialized (5) 822 → 606 (-26%) 14.02 → 14.25
Live 2.109 GB drain (5) 770 → 557 (-28%) 15.96 → 11.32
Live 5 × 422 MB drains (13, pooled) 2568 → 1227 (-52%) 7.36 → 7.93
Live 216 MB materialized (5) 1192 → 1040 (-12.8%) 26.98 → 26.78

Independent repeats find no throughput change beyond noise: live concurrent elapsed p≈0.29, with client drain time also unchanged (p=0.92). The ~14 GB caps cluster in low-bandwidth periods and hit both revisions; no progressive code-related slowdown reproduces. This is a peak-RSS improvement, not a universal speedup. V8-256 materialization OOMs and largest-materialization caps remain; historical worker results and main-to-stack context are retained in the handoff.

Adversarial review: round 1 converged with no open high finding; memory gains reproduced, throughput is within noise, and the path/error-listener coverage nits are fixed.

This pull request and its description were written by Isaac.

@vuanhphung
vuanhphung changed the base branch from main to perf/thrift-cloudfetch November 19, 2026 17:14
@vuanhphung
vuanhphung marked this pull request as ready for review November 19, 2026 18:59

@peco-review-bot peco-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔀 Cross-reviewed: two independent reviews (Claude Opus 5.5 · GPT-6.1 Sol), checked by a judge model (Claude Sonnet 5.5).

Verdict: 1 Nit

Looks good, no blocking concerns. node-fetch v2 installs the data/end/error reader listeners synchronously inside buffer(), so the before/after listener diff removes exactly those, and the constructor's own error handler is kept for late stream errors. The tests cover the success, rejection and production HTTP paths.

Other findings

  • ⚪ Nit — The new Unreleased CHANGELOG entry is a bare paragraph, but the entries after it are - bullets. Prefix it with - so the list stays consistent. (Found by Claude Opus 5.5 · confirmed by the judge)

@vuanhphung

Copy link
Copy Markdown
Collaborator Author

Fixed the changelog-list nit in e44f0af: the new Unreleased entry now uses the existing bullet style. npm run prettier passes.

@peco-review-bot peco-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔀 Cross-reviewed: two independent reviews (Claude Opus 5.5 · GPT-6.1 Sol), checked by a judge model (Claude Sonnet 5.5).

✅ No issues identified.

vuanhphung and others added 3 commits November 20, 2026 04:08
Detach only the listeners installed by node-fetch after each body read settles, preserving other listeners and late-error handling.

Signed-off-by: Vu Anh Phung <vu.phung@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
Signed-off-by: Vu Anh Phung <vu.phung@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
Signed-off-by: Vu Anh Phung <vu.phung@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
@vuanhphung
vuanhphung changed the base branch from perf/thrift-cloudfetch to main November 20, 2026 04:10

@peco-review-bot peco-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔀 Cross-reviewed: two independent reviews (Claude Opus 5.5 · GPT-6.1 Sol), checked by a judge model (Claude Sonnet 5.5).

✅ No issues identified.

Resolve the main changes by merging (no force-push).

Signed-off-by: Vu Anh Phung <vu.phung@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>

@peco-review-bot peco-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔀 Cross-reviewed: two independent reviews (Claude Opus 5.5 · GPT-6.1 Sol), checked by a judge model (Claude Sonnet 5.5).

✅ No issues identified.

Explain the Node core retention the listener cleanup works around: on a
pooled TLS socket, https.Agent#createConnection listeners pin the first
ClientRequest and, through node-fetch, the body readers and settled body.
Add an HTTPS keep-alive test on the production download path that asserts
the downloaded buffer is collectable (WeakRef + gc); it fails without the
cleanup, while plain HTTP retains nothing either way. Share the per-run
localhost certificate fixture with the HttpConnection proxy tests.

Co-authored-by: Isaac <no-reply@databricks.com>
Signed-off-by: Vu Anh Phung <vu.phung@databricks.com>

@peco-review-bot peco-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔀 Cross-reviewed: two independent reviews (Claude Opus 5.5 · GPT-6.1 Sol), checked by a judge model (Claude Sonnet 5.5).

✅ No issues identified.

This branch was previously deployed

1 inactive deployment
azure-prod — 0173b056 Deployed Nov 20, 2026 by vuanhphung via e2e-test (26.12.1, 26_12_1) #2189
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant