Repository navigation
Complete configured Volume transfers through the native backend - #613
aakash-saravanan-db wants to merge 43 commits into
Conversation
|
Thanks for your contribution! To satisfy the DCO policy in our contributing guide every commit message must include a sign-off message. One or more of your commits is missing this message. You can reword previous commit messages with an interactive rebase ( |
ad49367 to
7b276d7
Compare
7b276d7 to
12876fc
Compare
There was a problem hiding this comment.
Verdict: 1 Medium · 1 Low
Mostly sound, with one medium issue: on the kernel path, the new StagingError wrapping can also catch the statement's own execution errors, not just transfer errors. Nits: there is a stray blank line in the CHANGELOG entry, and the constructor comment in KernelOperationBackend still says cancel() "always routes to the cancellable execution". Follow-up (not caused by this PR): executeStatement waits for the transfer before it returns the operation, so a caller can only interrupt an in-flight transfer by closing the session.
Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>
Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>
Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>
Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>
Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>
Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>
Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>
Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>
Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>
Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>
Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>
Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>
Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>
Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>
Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>
Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>
Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>
Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>
Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>
Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>
Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>
Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>
Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>
There was a problem hiding this comment.
🔀 Cross-reviewed: two independent reviews (Claude Opus 5.5 · GPT-6.1 Sol), checked by a judge model (Claude Sonnet 5.5).
Verdict: 1 High · 2 Medium · 2 Low
Needs changes: 3 medium and 1 low concern. The staging dispatch and error mapping look right, but the PR ships behavior that depends on the unreleased native binding with no capability gate, and it changes socketTimeout: 0 semantics without a test or a CHANGELOG entry.
This overview is Claude Opus 5.5's initial take; GPT-6.1 Sol added findings it doesn't mention — see the findings below.
| const handle = await this.getFetchHandle(); | ||
| let outcome; | ||
| try { | ||
| outcome = await handle.completeVolumeOperation(allowedPaths, true); // Thrift REMOVE treats missing objects as success. |
There was a problem hiding this comment.
🟠 High — completeStagingOperation calls handle.completeVolumeOperation unconditionally, and assertBindingShape never checks for it. With an older installed binding (platform packages are still pinned to 1.4.0), kernel executeStatement with stagingAllowedLocalPath throws a raw TypeError. Previously it warned and returned results. Add a capability check that gives a clear error, and bump the pins together with this change.
🔀 Found by both reviewers · confirmed by the judge
| - Kernel backend: a failed status poll while waiting for a statement (a `runAsync: true` statement, or a default execution that outlives the server's inline wait) and a failed result fetch in `hasMoreRows()` now throw the mapped driver error, such as `HiveDriverError: HTTP 500: Internal Server Error` once retries on HTTP 5xx run out, instead of a raw `Error` whose message is the internal `__databricks_error__:{...}` envelope. As in `fetchChunk()`, a failed fetch in `hasMoreRows()` also closes the statement. (#608) | ||
| - Honor `cloudFetchConcurrentDownloads` when opening kernel sessions, with a default window of 16 and explicit chunk-setting overrides. Values other than positive safe integers raise `HiveDriverError` at session open. | ||
| - Reduce result chunking and `fetchAll()` overhead for large query results. | ||
| - Support Volume file transfers on the kernel backend when `stagingAllowedLocalPath` is supplied. Symlink targets must remain within configured allowed roots; in-root symlinks are permitted. |
There was a problem hiding this comment.
🔵 Low — The changelog entry doesn't say this needs a newer native kernel package than the pinned 1.4.0, and it doesn't reference the kernel change. Add the required binding version and the kernel PR.
🔀 Found by Claude Opus 5.5 · confirmed by the judge
| import DBSQLOperation from '../../../lib/DBSQLOperation'; | ||
| import asyncDisposable from '../../../lib/asyncDisposable'; | ||
| import { KernelNativeBinding, KernelConnection, KernelStatement } from '../../../lib/kernel/KernelNativeLoader'; | ||
| import { VolumeCompletion, openSession } from '../../../native/kernel'; |
There was a problem hiding this comment.
🟡 Medium — The eager runtime import of the real native module, plus a new test that opens a real native session, makes the mocked execution suite fail where the binding is absent or Node is older. Move the native test to the integration suite, and use string literals or a stub in the unit tests.
🔀 Found by both reviewers · confirmed by the judge
| ? options.stagingAllowedLocalPath | ||
| : [options.stagingAllowedLocalPath]; | ||
| if (opBackend.completeStagingOperation) { | ||
| await this.runBackend(() => opBackend.completeStagingOperation!(allowedLocalPath)); |
There was a problem hiding this comment.
🔵 Low — If native completion rejects, executeStatement throws after the operation was registered. The operation, and its native statement, stays open until the session closes. The Thrift staging path behaves the same way. Best-effort close the operation on failure and rethrow the original error.
🔀 Found by GPT-6.1 Sol · confirmed by the judge
| // default — preserving the "effectively no idle timeout" semantics. | ||
| if (typeof socketTimeout === 'number' && socketTimeout > 0) { | ||
| // Preserve zero so native storage inactivity can be disabled like Thrift. | ||
| if (typeof socketTimeout === 'number' && socketTimeout >= 0) { |
There was a problem hiding this comment.
🟡 Medium — socketTimeout: 0 is now forwarded as socketTimeoutMs: 0. Bindings that still treat the value as the reqwest total timeout (the currently pinned ones) would time out immediately. The change has no unit test. Gate it on the coordinated binding release, add tests for 0 and positive values, and add a changelog note.
🔀 Found by Claude Opus 5.5 · confirmed by the judge
Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>
There was a problem hiding this comment.
🔀 Cross-reviewed: two independent reviews (Claude Opus 5.5 · GPT-6.1 Sol), checked by a judge model (Claude Sonnet 5.5).
Verdict: 1 High · 1 Medium · 2 Low
Mostly sound, with 2 medium concerns. The staging dispatch and error mapping look correct. Merging is still blocked, as the PR description says, on the platform package pin and lockfile updates.
This overview is Claude Opus 5.5's initial take; GPT-6.1 Sol added findings it doesn't mention — see the findings below.
Other findings
- 🟠 High — Completion now unconditionally requires the new native API, but npm installations still receive the pinned 1.4.0 packages rather than the coordinated release required by this PR; updating KERNEL_REV only changes source builds. Update every platform package pin and lockfile to the compatible release and validate staging through a clean installed driver before merging. (Found by GPT-6.1 Sol · the judge couldn't confirm it)
| ? options.stagingAllowedLocalPath | ||
| : [options.stagingAllowedLocalPath]; | ||
| if (opBackend.completeStagingOperation) { | ||
| await this.runBackend(() => opBackend.completeStagingOperation!(allowedLocalPath)); |
There was a problem hiding this comment.
🔵 Low — The operation is registered by wrapOperation before completion runs. If completion throws, the caller never gets a handle and the statement stays open until session close. Best-effort close the operation when completion fails, and rethrow the original error.
🔀 Found by GPT-6.1 Sol · confirmed by the judge
| import DBSQLOperation from '../../../lib/DBSQLOperation'; | ||
| import asyncDisposable from '../../../lib/asyncDisposable'; | ||
| import { KernelNativeBinding, KernelConnection, KernelStatement } from '../../../lib/kernel/KernelNativeLoader'; | ||
| import { VolumeCompletion, openSession } from '../../../native/kernel'; |
There was a problem hiding this comment.
🟡 Medium — This value import loads the native binding when the test module loads. A missing or mismatched binding then breaks the whole execution.test.ts suite. Use import type and the 'not_volume' literal, and move the live openSession test to a suite that skips without the binding.
🔀 Found by Claude Opus 5.5 · confirmed by the judge
| // default — preserving the "effectively no idle timeout" semantics. | ||
| if (typeof socketTimeout === 'number' && socketTimeout > 0) { | ||
| // Preserve zero so native storage inactivity can be disabled like Thrift. | ||
| if (typeof socketTimeout === 'number' && socketTimeout >= 0) { |
There was a problem hiding this comment.
🔵 Low — socketTimeout 0 is now forwarded to the binding, and positive values also set the storage idle timeout. There is no unit test for buildKernelHttpOptions covering this, and the CHANGELOG has no entry for it. Add tests for 0 and positive values, and a CHANGELOG note that it needs the new native release.
🔀 Found by Claude Opus 5.5 · confirmed by the judge
Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>
There was a problem hiding this comment.
🔀 Cross-reviewed: two independent reviews (Claude Opus 5.5 · GPT-6.1 Sol), checked by a judge model (Claude Sonnet 5.5).
Verdict: 1 High · 1 Medium · 2 Low
Mostly sound: 1 medium and 1 low concern. The PR already notes that native pins are pending. Until they land, an older binding has no completeVolumeOperation, so every statement run with stagingAllowedLocalPath would throw a TypeError on the kernel backend instead of the old no-op.
This overview is Claude Opus 5.5's initial take; GPT-6.1 Sol added findings it doesn't mention — see the findings below.
| (mapped instanceof ParameterError && !metadata?.httpStatus) || | ||
| metadata?.errorCode === 'VOLUME_STORAGE_HTTP_ERROR' | ||
| ) { | ||
| const staging = new StagingError(mapped.message); |
There was a problem hiding this comment.
🔵 Low — Both the StagingError conversion and the local-file Error replacement drop the sqlState that mapKernelErrorToJsError attached, so SQLSTATE-based handling is lost for these failures. Re-attach sqlState with defineErrorMetadata on both replacements and assert it in tests.
🔀 Found by both reviewers · confirmed by the judge
| try { | ||
| outcome = await handle.completeVolumeOperation(allowedPaths, true); // Thrift REMOVE treats missing objects as success. | ||
| } catch (err) { | ||
| throw toStagingTransferError(decodeNapiKernelError(err)); |
There was a problem hiding this comment.
🔵 Low — When completeVolumeOperation fails, or needs_input_stream is returned, the error is thrown without closing the native statement. The caller never gets the operation handle, so it stays open until the session closes. Best-effort close the statement on failure without masking the original error.
🔀 Found by GPT-6.1 Sol · confirmed by the judge
| // default — preserving the "effectively no idle timeout" semantics. | ||
| if (typeof socketTimeout === 'number' && socketTimeout > 0) { | ||
| // Preserve zero so native storage inactivity can be disabled like Thrift. | ||
| if (typeof socketTimeout === 'number' && socketTimeout >= 0) { |
There was a problem hiding this comment.
🟡 Medium — socketTimeout: 0 is now forwarded to the native layer. A pre-change binding would treat 0 as an immediate reqwest timeout, and no test covers 0 or positive values. Add buildKernelHttpOptions tests and tie this change to the coordinated native release.
🔀 Found by Claude Opus 5.5 · confirmed by the judge
| const handle = await this.getFetchHandle(); | ||
| let outcome; | ||
| try { | ||
| outcome = await handle.completeVolumeOperation(allowedPaths, true); // Thrift REMOVE treats missing objects as success. |
There was a problem hiding this comment.
🟠 High — The kernel backend calls completeVolumeOperation, which exists only in the source-pinned KERNEL_REV. The published optionalDependencies are still pinned to 1.4.0, so npm consumers get a binding without it and hit a TypeError. Bump all platform pins and the lockfile to a compatible release, or add a capability check, before merging.
🔀 Found by GPT-6.1 Sol · confirmed by the judge
Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>
There was a problem hiding this comment.
🔀 Cross-reviewed: two independent reviews (Claude Opus 5.5 · GPT-6.1 Sol), checked by a judge model (Claude Sonnet 5.5).
Verdict: 1 High · 1 Medium · 2 Low
Needs changes: 1 high, 1 medium. With the currently pinned 1.4.0 native binaries, the staging completion call and the forwarded socketTimeout: 0 would break kernel users until the compatible binding pins land in this PR.
| const handle = await this.getFetchHandle(); | ||
| let outcome; | ||
| try { | ||
| outcome = await handle.completeVolumeOperation(allowedPaths, true); // Thrift REMOVE treats missing objects as success. |
There was a problem hiding this comment.
🟠 High — The new code calls native completeVolumeOperation for every kernel statement run with stagingAllowedLocalPath. package.json still pins all kernel platform packages at 1.4.0, which lack this method, so those statements fail with a raw TypeError. Bump all platform pins and lockfiles together, or feature-detect the method and throw a mapped error.
🔀 Found by both reviewers · confirmed by the judge
| - Kernel backend: a failed status poll while waiting for a statement (a `runAsync: true` statement, or a default execution that outlives the server's inline wait) and a failed result fetch in `hasMoreRows()` now throw the mapped driver error, such as `HiveDriverError: HTTP 500: Internal Server Error` once retries on HTTP 5xx run out, instead of a raw `Error` whose message is the internal `__databricks_error__:{...}` envelope. As in `fetchChunk()`, a failed fetch in `hasMoreRows()` also closes the statement. (#608) | ||
| - Honor `cloudFetchConcurrentDownloads` when opening kernel sessions, with a default window of 16 and explicit chunk-setting overrides. Values other than positive safe integers raise `HiveDriverError` at session open. | ||
| - Reduce result chunking and `fetchAll()` overhead for large query results. | ||
| - Support Volume file transfers on the kernel backend when `stagingAllowedLocalPath` is supplied. Symlink targets must remain within configured allowed roots; in-root symlinks are permitted. (#613) |
There was a problem hiding this comment.
🔵 Low — The changelog entry doesn't say a published kernel package bump is required. It also doesn't mention that socketTimeout: 0 is now forwarded and that the timeout now bounds storage idle time. Add both notes as separate entries.
🔀 Found by Claude Opus 5.5 · confirmed by the judge
| (mapped instanceof ParameterError && !metadata?.httpStatus) || | ||
| metadata?.errorCode === 'VOLUME_STORAGE_HTTP_ERROR' | ||
| ) { | ||
| const staging = new StagingError(mapped.message); |
There was a problem hiding this comment.
🔵 Low — The StagingError and local-file replacement errors drop the sqlState decoded from the envelope, unlike other mapped kernel errors. Re-attach sqlState with defineErrorMetadata in both paths and add test assertions.
🔀 Found by both reviewers · confirmed by the judge
| // default — preserving the "effectively no idle timeout" semantics. | ||
| if (typeof socketTimeout === 'number' && socketTimeout > 0) { | ||
| // Preserve zero so native storage inactivity can be disabled like Thrift. | ||
| if (typeof socketTimeout === 'number' && socketTimeout >= 0) { |
There was a problem hiding this comment.
🟡 Medium — socketTimeout: 0 is now forwarded as socketTimeoutMs: 0. The old comment said that value made reqwest time out immediately, and only the new native core treats it as disabling storage inactivity. With the 1.4.0 pins it can break requests, so omit zero until the compatible binding is pinned, or gate on binding capability.
🔀 Found by Claude Opus 5.5 · confirmed by the judge
Feature or gap
Configured staging operations need to finish their Volume transfer through the native backend before executeStatement resolves.
What this PR addresses
Delegate marked file PUT, GET and REMOVE to the shared native core, preserve raw results when staging is omitted, and map staging and local-file errors through the Node API. Native completion declarations and enum exports are generated from the binding source.
Dependencies
Requires the Volume APIs in databricks/databricks-sql-kernel#465 and a compatible coordinated native release. Platform package pins and locks, plus clean installed-driver validation, must be updated before merge.
Kernel version
The compatible kernel release has not been selected. When it is available, update every platform package pin and lockfile in this PR and validate a clean installation before merge.
KERNEL_REVselects source for local and CI builds only.Tests and validation
The focused suites pass: 306 tests. TypeScript build, production ESLint and formatting pass. Native declarations use NAPI-RS CLI 3.8.2.