Skip to content

Complete configured Volume transfers through the native backend - #613

Open
aakash-saravanan-db wants to merge 43 commits into
mainfrom
aakash-saravanan_data/feat/volume-restart-20261113
Open

aakash-saravanan-db wants to merge 43 commits into
mainfrom
aakash-saravanan_data/feat/volume-restart-20261113

Conversation

@aakash-saravanan-db

@aakash-saravanan-db aakash-saravanan-db commented Nov 14, 2026 •

Copy link
Copy Markdown
Collaborator

Feature or gap

Configured staging operations need to finish their Volume transfer through the native backend before executeStatement resolves.

What this PR addresses

Delegate marked file PUT, GET and REMOVE to the shared native core, preserve raw results when staging is omitted, and map staging and local-file errors through the Node API. Native completion declarations and enum exports are generated from the binding source.

Dependencies

Requires the Volume APIs in databricks/databricks-sql-kernel#465 and a compatible coordinated native release. Platform package pins and locks, plus clean installed-driver validation, must be updated before merge.

Kernel version

The compatible kernel release has not been selected. When it is available, update every platform package pin and lockfile in this PR and validate a clean installation before merge. KERNEL_REV selects source for local and CI builds only.

Tests and validation

The focused suites pass: 306 tests. TypeScript build, production ESLint and formatting pass. Native declarations use NAPI-RS CLI 3.8.2.

@github-actions

Copy link
Copy Markdown

Thanks for your contribution! To satisfy the DCO policy in our contributing guide every commit message must include a sign-off message. One or more of your commits is missing this message. You can reword previous commit messages with an interactive rebase (git rebase -i main).

@aakash-saravanan-db
aakash-saravanan-db force-pushed the aakash-saravanan_data/feat/volume-restart-20261113 branch from ad49367 to 7b276d7 Compare November 14, 2026 00:40
@github-actions github-actions Bot removed the kernel-e2e Preview-run the SEA Kernel E2E suite on this PR label Nov 14, 2026
@aakash-saravanan-db aakash-saravanan-db added the kernel-e2e Preview-run the SEA Kernel E2E suite on this PR label Nov 14, 2026
@aakash-saravanan-db
aakash-saravanan-db force-pushed the aakash-saravanan_data/feat/volume-restart-20261113 branch from 7b276d7 to 12876fc Compare November 14, 2026 00:52
@aakash-saravanan-db aakash-saravanan-db added kernel-e2e Preview-run the SEA Kernel E2E suite on this PR and removed kernel-e2e Preview-run the SEA Kernel E2E suite on this PR labels Nov 14, 2026
@aakash-saravanan-db
aakash-saravanan-db marked this pull request as ready for review November 14, 2026 02:18

@peco-review-bot peco-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: 1 Medium · 1 Low

Mostly sound, with one medium issue: on the kernel path, the new StagingError wrapping can also catch the statement's own execution errors, not just transfer errors. Nits: there is a stray blank line in the CHANGELOG entry, and the constructor comment in KernelOperationBackend still says cancel() "always routes to the cancellable execution". Follow-up (not caused by this PR): executeStatement waits for the transfer before it returns the operation, so a caller can only interrupt an in-flight transfer by closing the session.

Comment thread lib/kernel/KernelOperationBackend.ts Resolved
Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>
Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>
Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>
Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>
Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>
Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>
Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>
Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>
Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>
Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>
Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>
Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>
Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>
Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>
Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>
Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>
Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>
Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>
Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>
Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>
Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>
Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>
Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>

@peco-review-bot peco-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔀 Cross-reviewed: two independent reviews (Claude Opus 5.5 · GPT-6.1 Sol), checked by a judge model (Claude Sonnet 5.5).

Verdict: 1 High · 2 Medium · 2 Low

Needs changes: 3 medium and 1 low concern. The staging dispatch and error mapping look right, but the PR ships behavior that depends on the unreleased native binding with no capability gate, and it changes socketTimeout: 0 semantics without a test or a CHANGELOG entry.

This overview is Claude Opus 5.5's initial take; GPT-6.1 Sol added findings it doesn't mention — see the findings below.

const handle = await this.getFetchHandle();
let outcome;
try {
outcome = await handle.completeVolumeOperation(allowedPaths, true); // Thrift REMOVE treats missing objects as success.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟠 High — completeStagingOperation calls handle.completeVolumeOperation unconditionally, and assertBindingShape never checks for it. With an older installed binding (platform packages are still pinned to 1.4.0), kernel executeStatement with stagingAllowedLocalPath throws a raw TypeError. Previously it warned and returned results. Add a capability check that gives a clear error, and bump the pins together with this change.

🔀 Found by both reviewers · confirmed by the judge

Comment thread CHANGELOG.md Outdated
- Kernel backend: a failed status poll while waiting for a statement (a `runAsync: true` statement, or a default execution that outlives the server's inline wait) and a failed result fetch in `hasMoreRows()` now throw the mapped driver error, such as `HiveDriverError: HTTP 500: Internal Server Error` once retries on HTTP 5xx run out, instead of a raw `Error` whose message is the internal `__databricks_error__:{...}` envelope. As in `fetchChunk()`, a failed fetch in `hasMoreRows()` also closes the statement. (#608)
- Honor `cloudFetchConcurrentDownloads` when opening kernel sessions, with a default window of 16 and explicit chunk-setting overrides. Values other than positive safe integers raise `HiveDriverError` at session open.
- Reduce result chunking and `fetchAll()` overhead for large query results.
- Support Volume file transfers on the kernel backend when `stagingAllowedLocalPath` is supplied. Symlink targets must remain within configured allowed roots; in-root symlinks are permitted.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Low — The changelog entry doesn't say this needs a newer native kernel package than the pinned 1.4.0, and it doesn't reference the kernel change. Add the required binding version and the kernel PR.

🔀 Found by Claude Opus 5.5 · confirmed by the judge

Comment thread tests/unit/kernel/execution.test.ts Outdated
import DBSQLOperation from '../../../lib/DBSQLOperation';
import asyncDisposable from '../../../lib/asyncDisposable';
import { KernelNativeBinding, KernelConnection, KernelStatement } from '../../../lib/kernel/KernelNativeLoader';
import { VolumeCompletion, openSession } from '../../../native/kernel';

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Medium — The eager runtime import of the real native module, plus a new test that opens a real native session, makes the mocked execution suite fail where the binding is absent or Node is older. Move the native test to the integration suite, and use string literals or a stub in the unit tests.

🔀 Found by both reviewers · confirmed by the judge

Comment thread lib/DBSQLSession.ts
? options.stagingAllowedLocalPath
: [options.stagingAllowedLocalPath];
if (opBackend.completeStagingOperation) {
await this.runBackend(() => opBackend.completeStagingOperation!(allowedLocalPath));

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Low — If native completion rejects, executeStatement throws after the operation was registered. The operation, and its native statement, stays open until the session closes. The Thrift staging path behaves the same way. Best-effort close the operation on failure and rethrow the original error.

🔀 Found by GPT-6.1 Sol · confirmed by the judge

Comment thread lib/kernel/KernelAuth.ts
// default — preserving the "effectively no idle timeout" semantics.
if (typeof socketTimeout === 'number' && socketTimeout > 0) {
// Preserve zero so native storage inactivity can be disabled like Thrift.
if (typeof socketTimeout === 'number' && socketTimeout >= 0) {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Medium — socketTimeout: 0 is now forwarded as socketTimeoutMs: 0. Bindings that still treat the value as the reqwest total timeout (the currently pinned ones) would time out immediately. The change has no unit test. Gate it on the coordinated binding release, add tests for 0 and positive values, and add a changelog note.

🔀 Found by Claude Opus 5.5 · confirmed by the judge

Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>

@peco-review-bot peco-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔀 Cross-reviewed: two independent reviews (Claude Opus 5.5 · GPT-6.1 Sol), checked by a judge model (Claude Sonnet 5.5).

Verdict: 1 High · 1 Medium · 2 Low

Mostly sound, with 2 medium concerns. The staging dispatch and error mapping look correct. Merging is still blocked, as the PR description says, on the platform package pin and lockfile updates.

This overview is Claude Opus 5.5's initial take; GPT-6.1 Sol added findings it doesn't mention — see the findings below.

Other findings

  • 🟠 High — Completion now unconditionally requires the new native API, but npm installations still receive the pinned 1.4.0 packages rather than the coordinated release required by this PR; updating KERNEL_REV only changes source builds. Update every platform package pin and lockfile to the compatible release and validate staging through a clean installed driver before merging. (Found by GPT-6.1 Sol · the judge couldn't confirm it)

Comment thread lib/DBSQLSession.ts
? options.stagingAllowedLocalPath
: [options.stagingAllowedLocalPath];
if (opBackend.completeStagingOperation) {
await this.runBackend(() => opBackend.completeStagingOperation!(allowedLocalPath));

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Low — The operation is registered by wrapOperation before completion runs. If completion throws, the caller never gets a handle and the statement stays open until session close. Best-effort close the operation when completion fails, and rethrow the original error.

🔀 Found by GPT-6.1 Sol · confirmed by the judge

Comment thread tests/unit/kernel/execution.test.ts Outdated
import DBSQLOperation from '../../../lib/DBSQLOperation';
import asyncDisposable from '../../../lib/asyncDisposable';
import { KernelNativeBinding, KernelConnection, KernelStatement } from '../../../lib/kernel/KernelNativeLoader';
import { VolumeCompletion, openSession } from '../../../native/kernel';

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Medium — This value import loads the native binding when the test module loads. A missing or mismatched binding then breaks the whole execution.test.ts suite. Use import type and the 'not_volume' literal, and move the live openSession test to a suite that skips without the binding.

🔀 Found by Claude Opus 5.5 · confirmed by the judge

Comment thread lib/kernel/KernelAuth.ts
// default — preserving the "effectively no idle timeout" semantics.
if (typeof socketTimeout === 'number' && socketTimeout > 0) {
// Preserve zero so native storage inactivity can be disabled like Thrift.
if (typeof socketTimeout === 'number' && socketTimeout >= 0) {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Low — socketTimeout 0 is now forwarded to the binding, and positive values also set the storage idle timeout. There is no unit test for buildKernelHttpOptions covering this, and the CHANGELOG has no entry for it. Add tests for 0 and positive values, and a CHANGELOG note that it needs the new native release.

🔀 Found by Claude Opus 5.5 · confirmed by the judge

Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>

@peco-review-bot peco-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔀 Cross-reviewed: two independent reviews (Claude Opus 5.5 · GPT-6.1 Sol), checked by a judge model (Claude Sonnet 5.5).

Verdict: 1 High · 1 Medium · 2 Low

Mostly sound: 1 medium and 1 low concern. The PR already notes that native pins are pending. Until they land, an older binding has no completeVolumeOperation, so every statement run with stagingAllowedLocalPath would throw a TypeError on the kernel backend instead of the old no-op.

This overview is Claude Opus 5.5's initial take; GPT-6.1 Sol added findings it doesn't mention — see the findings below.

(mapped instanceof ParameterError && !metadata?.httpStatus) ||
metadata?.errorCode === 'VOLUME_STORAGE_HTTP_ERROR'
) {
const staging = new StagingError(mapped.message);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Low — Both the StagingError conversion and the local-file Error replacement drop the sqlState that mapKernelErrorToJsError attached, so SQLSTATE-based handling is lost for these failures. Re-attach sqlState with defineErrorMetadata on both replacements and assert it in tests.

🔀 Found by both reviewers · confirmed by the judge

try {
outcome = await handle.completeVolumeOperation(allowedPaths, true); // Thrift REMOVE treats missing objects as success.
} catch (err) {
throw toStagingTransferError(decodeNapiKernelError(err));

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Low — When completeVolumeOperation fails, or needs_input_stream is returned, the error is thrown without closing the native statement. The caller never gets the operation handle, so it stays open until the session closes. Best-effort close the statement on failure without masking the original error.

🔀 Found by GPT-6.1 Sol · confirmed by the judge

Comment thread lib/kernel/KernelAuth.ts
// default — preserving the "effectively no idle timeout" semantics.
if (typeof socketTimeout === 'number' && socketTimeout > 0) {
// Preserve zero so native storage inactivity can be disabled like Thrift.
if (typeof socketTimeout === 'number' && socketTimeout >= 0) {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Medium — socketTimeout: 0 is now forwarded to the native layer. A pre-change binding would treat 0 as an immediate reqwest timeout, and no test covers 0 or positive values. Add buildKernelHttpOptions tests and tie this change to the coordinated native release.

🔀 Found by Claude Opus 5.5 · confirmed by the judge

const handle = await this.getFetchHandle();
let outcome;
try {
outcome = await handle.completeVolumeOperation(allowedPaths, true); // Thrift REMOVE treats missing objects as success.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟠 High — The kernel backend calls completeVolumeOperation, which exists only in the source-pinned KERNEL_REV. The published optionalDependencies are still pinned to 1.4.0, so npm consumers get a binding without it and hit a TypeError. Bump all platform pins and the lockfile to a compatible release, or add a capability check, before merging.

🔀 Found by GPT-6.1 Sol · confirmed by the judge

Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>

@peco-review-bot peco-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔀 Cross-reviewed: two independent reviews (Claude Opus 5.5 · GPT-6.1 Sol), checked by a judge model (Claude Sonnet 5.5).

Verdict: 1 High · 1 Medium · 2 Low

Needs changes: 1 high, 1 medium. With the currently pinned 1.4.0 native binaries, the staging completion call and the forwarded socketTimeout: 0 would break kernel users until the compatible binding pins land in this PR.

const handle = await this.getFetchHandle();
let outcome;
try {
outcome = await handle.completeVolumeOperation(allowedPaths, true); // Thrift REMOVE treats missing objects as success.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟠 High — The new code calls native completeVolumeOperation for every kernel statement run with stagingAllowedLocalPath. package.json still pins all kernel platform packages at 1.4.0, which lack this method, so those statements fail with a raw TypeError. Bump all platform pins and lockfiles together, or feature-detect the method and throw a mapped error.

🔀 Found by both reviewers · confirmed by the judge

Comment thread CHANGELOG.md
- Kernel backend: a failed status poll while waiting for a statement (a `runAsync: true` statement, or a default execution that outlives the server's inline wait) and a failed result fetch in `hasMoreRows()` now throw the mapped driver error, such as `HiveDriverError: HTTP 500: Internal Server Error` once retries on HTTP 5xx run out, instead of a raw `Error` whose message is the internal `__databricks_error__:{...}` envelope. As in `fetchChunk()`, a failed fetch in `hasMoreRows()` also closes the statement. (#608)
- Honor `cloudFetchConcurrentDownloads` when opening kernel sessions, with a default window of 16 and explicit chunk-setting overrides. Values other than positive safe integers raise `HiveDriverError` at session open.
- Reduce result chunking and `fetchAll()` overhead for large query results.
- Support Volume file transfers on the kernel backend when `stagingAllowedLocalPath` is supplied. Symlink targets must remain within configured allowed roots; in-root symlinks are permitted. (#613)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Low — The changelog entry doesn't say a published kernel package bump is required. It also doesn't mention that socketTimeout: 0 is now forwarded and that the timeout now bounds storage idle time. Add both notes as separate entries.

🔀 Found by Claude Opus 5.5 · confirmed by the judge

(mapped instanceof ParameterError && !metadata?.httpStatus) ||
metadata?.errorCode === 'VOLUME_STORAGE_HTTP_ERROR'
) {
const staging = new StagingError(mapped.message);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Low — The StagingError and local-file replacement errors drop the sqlState decoded from the envelope, unlike other mapped kernel errors. Re-attach sqlState with defineErrorMetadata in both paths and add test assertions.

🔀 Found by both reviewers · confirmed by the judge

Comment thread lib/kernel/KernelAuth.ts
// default — preserving the "effectively no idle timeout" semantics.
if (typeof socketTimeout === 'number' && socketTimeout > 0) {
// Preserve zero so native storage inactivity can be disabled like Thrift.
if (typeof socketTimeout === 'number' && socketTimeout >= 0) {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Medium — socketTimeout: 0 is now forwarded as socketTimeoutMs: 0. The old comment said that value made reqwest time out immediately, and only the new native core treats it as disabling storage inactivity. With the 1.4.0 pins it can break requests, so omit zero until the compatible binding is pinned, or gate on binding capability.

🔀 Found by Claude Opus 5.5 · confirmed by the judge

This branch was previously deployed

1 inactive deployment
azure-prod — 113f7ad1 Deployed Nov 20, 2026 by aakash-saravanan-db via e2e-test (26.12.1, 26_12_1) #2195
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant