Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
43 commits
Select commit Hold shift + click to select a range
836b8d8
Complete configured Volume operations in the native backend
aakash-saravanan-db Nov 13, 2026
49f32ae
Retain inline results for configured Volume transfers
aakash-saravanan-db Nov 13, 2026
531e8a0
Correct Volume backend support documentation
aakash-saravanan-db Nov 13, 2026
bd3d68c
Pin the native source for Volume completion
aakash-saravanan-db Nov 14, 2026
3a01877
Select the Volume kernel and format its path option
aakash-saravanan-db Nov 14, 2026
fc0ac1d
Regenerate Volume completion declarations from the pinned kernel
aakash-saravanan-db Nov 14, 2026
b7e0be6
Preserve SQL execution errors during Volume completion
aakash-saravanan-db Nov 14, 2026
3c6a8cc
Pin native Volume storage retry and terminal response fixes
aakash-saravanan-db Nov 14, 2026
0f5be1e
Cover portable native local file error decoding
aakash-saravanan-db Nov 14, 2026
4bf828a
Preserve deferred Volume instruction HTTP errors
aakash-saravanan-db Nov 14, 2026
381f4e0
Preserve portable file error fallback for unknown errno values
aakash-saravanan-db Nov 14, 2026
f8df87c
build(kernel): select corrected Volume completion
aakash-saravanan-db Nov 14, 2026
d58e929
build: select native Volume cancellation and error handling
aakash-saravanan-db Nov 15, 2026
191582b
build: use native root authorization fix
aakash-saravanan-db Nov 15, 2026
0d84261
build: update kernel source for replayable uploads
aakash-saravanan-db Nov 15, 2026
2e2c4c7
build: update kernel for local stream close
aakash-saravanan-db Nov 15, 2026
745f36f
fix(volume): select replayable file uploads and tighten validation
aakash-saravanan-db Nov 15, 2026
efc0e10
fix(kernel): use permission-preserving Volume downloads
aakash-saravanan-db Nov 16, 2026
a4f1649
build(kernel): select sanitized storage redirects
aakash-saravanan-db Nov 16, 2026
11b1d72
Clarify Volume cancellation and consolidate routing coverage
aakash-saravanan-db Nov 16, 2026
6be0b76
Validate native file error numbers and select shared completion
aakash-saravanan-db Nov 16, 2026
7537263
Pin kernel with statement lifecycle cleanup
aakash-saravanan-db Nov 16, 2026
0a266a2
build: select kernel with async result cleanup
aakash-saravanan-db Nov 16, 2026
9003e80
build: select kernel with Volume instruction error telemetry
aakash-saravanan-db Nov 16, 2026
3bf5367
build: pin kernel with complete Volume instruction handling
aakash-saravanan-db Nov 16, 2026
86319e7
fix(kernel): preserve ordinary post-SQL cancellation routing
aakash-saravanan-db Nov 17, 2026
7787206
build: use corrected kernel volume completion
aakash-saravanan-db Nov 17, 2026
9f511fb
build: select kernel with origin-safe Volume redirects
aakash-saravanan-db Nov 17, 2026
dea2867
build: select kernel with normalized upload lengths
aakash-saravanan-db Nov 17, 2026
08d2e83
docs: keep unreleased changelog entries in one list
aakash-saravanan-db Nov 17, 2026
5e80236
build: pin kernel completion telemetry correction
aakash-saravanan-db Nov 17, 2026
2de1567
build: select kernel with restricted-directory GET support
aakash-saravanan-db Nov 18, 2026
c960aa4
fix(kernel): cancel pending result waits before close
aakash-saravanan-db Nov 18, 2026
e0d4aaf
fix(kernel): await shared statement disposal on close
aakash-saravanan-db Nov 18, 2026
c842040
Finalize concurrent operation cleanup once
aakash-saravanan-db Nov 18, 2026
c0e8f45
build: pin kernel with bounded symlink resolution
aakash-saravanan-db Nov 18, 2026
8f7882e
refactor(volume): reduce staging completion plumbing
aakash-saravanan-db Nov 19, 2026
e2611e0
build: select updated Volume kernel
aakash-saravanan-db Nov 19, 2026
816d808
fix(volume): poll staged statements before native completion
aakash-saravanan-db Nov 20, 2026
40c34bf
fixup! fix(volume): poll staged statements before native completion
aakash-saravanan-db Nov 20, 2026
e48dfc9
fixup! fix(volume): poll staged statements before native completion
aakash-saravanan-db Nov 20, 2026
1d03dbe
fixup! fix(volume): poll staged statements before native completion
aakash-saravanan-db Nov 20, 2026
113f7ad
fixup! fix(volume): poll staged statements before native completion
aakash-saravanan-db Nov 20, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/kernel-e2e.yml
Original file line number Diff line number Diff line change
Expand Up @@ -160,7 +160,7 @@ jobs:
# Run when the kernel driver layer, the napi binding contract, kernel
# e2e tests, this workflow, the kernel revision pin, or core deps
# move.
if echo "$CHANGED" | grep -qE "^(lib/kernel/|lib/result/|native/kernel/|tests/e2e/kernel/|tests/e2e/utils/rowMaterialization\.ts|tests/unit/kernel/|\.github/workflows/kernel-e2e\.yml|KERNEL_REV|package\.json|package-lock\.json)"; then
if echo "$CHANGED" | grep -qE "^(lib/kernel/|lib/result/|native/kernel/|tests/e2e/kernel/|tests/e2e/utils/rowMaterialization\.ts|tests/e2e/utils/stagingTests\.ts|tests/unit/kernel/|\.github/workflows/kernel-e2e\.yml|KERNEL_REV|package\.json|package-lock\.json)"; then
echo "run_tests=true" >> "$GITHUB_OUTPUT"
else
echo "run_tests=false" >> "$GITHUB_OUTPUT"
Expand Down
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@
- Kernel backend: a failed status poll while waiting for a statement (a `runAsync: true` statement, or a default execution that outlives the server's inline wait) and a failed result fetch in `hasMoreRows()` now throw the mapped driver error, such as `HiveDriverError: HTTP 500: Internal Server Error` once retries on HTTP 5xx run out, instead of a raw `Error` whose message is the internal `__databricks_error__:{...}` envelope. As in `fetchChunk()`, a failed fetch in `hasMoreRows()` also closes the statement. (#608)
- Honor `cloudFetchConcurrentDownloads` when opening kernel sessions, with a default window of 16 and explicit chunk-setting overrides. Values other than positive safe integers raise `HiveDriverError` at session open.
- Reduce result chunking and `fetchAll()` overhead for large query results.
- Support Volume file transfers on the kernel backend when `stagingAllowedLocalPath` is supplied. Symlink targets must remain within configured allowed roots; in-root symlinks are permitted. (#613)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Low — The changelog entry doesn't say a published kernel package bump is required. It also doesn't mention that socketTimeout: 0 is now forwarded and that the timeout now bounds storage idle time. Add both notes as separate entries.

🔀 Found by Claude Opus 5.5 · confirmed by the judge

- Thrift CloudFetch downloads are now bounded end to end by `retriesTimeout` (default 15 minutes), including every attempt and retry backoff. Previously a stalled download, where the response headers or body never arrived, blocked the fetch forever. A download still running when the deadline expires is aborted and fails with `RetryError` (`TIMEOUT_EXCEEDED`, message `CloudFetch result download failed: Retry timeout exceeded`). (#520)
- Thrift connections made through `proxy` now apply `checkServerCertificate`, `customCaCert` and `clientCert`/`clientKey`. Previously the proxied path ignored them and verified against Node's process-wide trust store, so a certificate outside the configured trust set could be accepted. **Compatibility:** proxied connections now verify unless `checkServerCertificate: false` is set; as on direct connections, `NODE_TLS_REJECT_UNAUTHORIZED=0` no longer turns verification off. For an `https` proxy these options also apply to the proxy's own certificate. (#485)
- Honor the client `useCloudFetch: false` default on the kernel backend so complete results can be fetched without cloud-storage downloads. (#574)
Expand Down
27 changes: 13 additions & 14 deletions CONNECTION_PARAMETERS.md
Original file line number Diff line number Diff line change
Expand Up @@ -76,17 +76,17 @@ Per-statement `useCloudFetch` overrides remain unsupported on the kernel backend

## HTTP client, proxy, and retries

| Option | Type | Thrift | Kernel | Default Value | Note |
| --------------------------- | ------------------------ | :----: | :----: | ------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `socketTimeout` | `number` (ms) | ✅ | ⚠️ | `900000` / `120000` | Kernel forwards only positive values. `0` means no timeout on Thrift but is omitted on Kernel, which then keeps its 120-second default. |
| `proxy` | `ProxyOptions` | ✅ | ⚠️ | — | `{protocol, host, port, auth}`. Kernel supports HTTP/HTTPS proxies only; Thrift also supports the declared SOCKS variants. |
| `noProxy` (internal) | `string` | ❌ | ⚠️ | — | Runtime-only Kernel bypass list, forwarded as `bypassHosts` only when an explicit `proxy` is also supplied. |
| `customHeaders` | `Record<string, string>` | ✅ | ✅ | — | Sent on workspace, telemetry and feature-flag requests, not OAuth. Workspace calls ignore caller `Authorization`, `User-Agent`, `Content-Type`, `Content-Length`, `Transfer-Encoding`, `Connection`; Kernel also ignores `x-databricks-org-id`. |
| `retryMaxAttempts` | `number` | ✅ | ✅ | `5` | Total attempts, including the initial request. Kernel converts this to its retries-after-first representation internally. |
| `retriesTimeout` | `number` (ms) | ✅ | ✅ | `900000` | Kernel converts milliseconds to whole seconds. |
| `retryDelayMin` | `number` (ms) | ✅ | ✅ | `1000` | Kernel converts milliseconds to whole seconds. |
| `retryDelayMax` | `number` (ms) | ✅ | ✅ | `60000` | Kernel converts milliseconds to whole seconds. |
| `maxConnections` (internal) | `number` | ❌ | ✅ | — / Kernel default | Kernel connection-pool size. Must be a positive integer within the napi `u32` range. |
| Option | Type | Thrift | Kernel | Default Value | Note |
| --------------------------- | ------------------------ | :----: | :----: | ------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `socketTimeout` | `number` (ms) | ✅ | ⚠️ | `900000` / `120000` | Kernel uses a total SQL request deadline and a storage idle timeout (15 minutes by default). Positive values set both; `0` keeps the SQL default and disables storage inactivity. Storage idle time bounds reads, upload progress and PUT response waits; redirected stream PUT resends have no idle bound. |
| `proxy` | `ProxyOptions` | ✅ | ⚠️ | — | `{protocol, host, port, auth}`. Kernel supports HTTP/HTTPS proxies only; Thrift also supports the declared SOCKS variants. |
| `noProxy` (internal) | `string` | ❌ | ⚠️ | — | Runtime-only Kernel bypass list, forwarded as `bypassHosts` only when an explicit `proxy` is also supplied. |
| `customHeaders` | `Record<string, string>` | ✅ | ✅ | — | Sent on workspace, telemetry and feature-flag requests, not OAuth. Workspace calls ignore caller `Authorization`, `User-Agent`, `Content-Type`, `Content-Length`, `Transfer-Encoding`, `Connection`; Kernel also ignores `x-databricks-org-id`. |
| `retryMaxAttempts` | `number` | ✅ | ✅ | `5` | Total attempts, including the initial request. Kernel converts this to its retries-after-first representation internally. |
| `retriesTimeout` | `number` (ms) | ✅ | ✅ | `900000` | Kernel converts milliseconds to whole seconds. |
| `retryDelayMin` | `number` (ms) | ✅ | ✅ | `1000` | Kernel converts milliseconds to whole seconds. |
| `retryDelayMax` | `number` (ms) | ✅ | ✅ | `60000` | Kernel converts milliseconds to whole seconds. |
| `maxConnections` (internal) | `number` | ❌ | ✅ | — / Kernel default | Kernel connection-pool size. Must be a positive integer within the napi `u32` range. |

## TLS / SSL

Expand Down Expand Up @@ -213,7 +213,7 @@ telemetry implementation, so only the options explicitly forwarded by
| `maxRows` | `number \| bigint \| Int64 \| null` | ✅ | ❌ | `100000` / — | Thrift uses it for the initial Direct Results row count; `null` disables Direct Results. Kernel does not retain or forward the supplied value. |
| `useCloudFetch` | `boolean` | ✅ | ❌ | `true` / — | Kernel logs and ignores this per-statement hint; Kernel result fetching owns its CloudFetch behavior. |
| `useLZ4Compression` | `boolean` | ✅ | ❌ | `true` / — | Thrift uses it when supported and when the result is not CloudFetch. Kernel owns and auto-detects result compression. |
| `stagingAllowedLocalPath` | `string \| string[]` | ✅ | ❌ | — | Local allowlist for Thrift volume/staging operations. Kernel volume operations are unsupported. |
| `stagingAllowedLocalPath` | `string \| string[]` | ✅ | ✅ | — | Local allowlist for Volume operations. Kernel also requires resolved symlink targets within allowed roots; links within those roots are permitted. |
| `namedParameters` | `Record<string, DBSQLParameter \| DBSQLParameterValue>` | ✅ | ✅ | — | Named SQL parameters. Cannot be combined with non-empty `ordinalParameters`. Thrift requires a protocol that supports parameterized queries. |
| `ordinalParameters` | `Array<DBSQLParameter \| DBSQLParameterValue>` | ✅ | ✅ | — | Positional SQL parameters. Cannot be combined with non-empty `namedParameters`. |
| `queryTags` | `Record<string, string \| null \| undefined>` | ✅ | ✅ | — | Serialized into the per-statement `query_tags` conf overlay on both. |
Expand Down Expand Up @@ -244,8 +244,7 @@ session parameter surface.

Thrift-only capabilities include custom/token-provider authentication, custom
OAuth persistence, SOCKS proxies, authenticated/driver-buffer telemetry knobs,
query timeout, Direct Results `maxRows`, per-statement CloudFetch/LZ4 hints, and
volume operations.
query timeout, Direct Results `maxRows`, and per-statement CloudFetch/LZ4 hints.

Kernel-only capabilities include connection-pool sizing, proxy bypass hosts,
independent TLS hostname verification, optional U2M disk caching, and
Expand Down
2 changes: 1 addition & 1 deletion KERNEL_REV
Original file line number Diff line number Diff line change
@@ -1 +1 @@
dac32e2bf44468bdab06747dad4f289d1c0a18bd
c2ddddc61bd41d9529dd19f36546d603ba21150d
11 changes: 8 additions & 3 deletions lib/DBSQLSession.ts
Original file line number Diff line number Diff line change
Expand Up @@ -108,12 +108,17 @@ export default class DBSQLSession implements IDBSQLSession {
// If `stagingAllowedLocalPath` is not provided - don't do anything to the operation. In a case of regular
// operation, everything will work as usual. In a case of staging operation, it will be processed like any
// other query - it will be possible to get data from it as usual, or use other operation methods.
// Backends with explicit staging completion handle it before metadata is fetched.
if (options.stagingAllowedLocalPath !== undefined) {
const allowedLocalPath = Array.isArray(options.stagingAllowedLocalPath)
? options.stagingAllowedLocalPath
: [options.stagingAllowedLocalPath];
if (opBackend.completeStagingOperation) {
await this.runBackend(() => opBackend.completeStagingOperation!(allowedLocalPath));
Comment thread
aakash-saravanan-db marked this conversation as resolved.
Resolved
Comment thread
aakash-saravanan-db marked this conversation as resolved.
Resolved

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Low — If native completion rejects, executeStatement throws after the operation was registered. The operation, and its native statement, stays open until the session closes. The Thrift staging path behaves the same way. Best-effort close the operation on failure and rethrow the original error.

🔀 Found by GPT-6.1 Sol · confirmed by the judge

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Low — The operation is registered by wrapOperation before completion runs. If completion throws, the caller never gets a handle and the statement stays open until session close. Best-effort close the operation when completion fails, and rethrow the original error.

🔀 Found by GPT-6.1 Sol · confirmed by the judge

return operation;
}
const metadata = await operation.getResultMetadata();
if (metadata.isStagingOperation) {
const allowedLocalPath = Array.isArray(options.stagingAllowedLocalPath)
? options.stagingAllowedLocalPath
: [options.stagingAllowedLocalPath];
return this.handleStagingOperation(operation, allowedLocalPath, metadata);
}
}
Expand Down
3 changes: 2 additions & 1 deletion lib/contracts/IDBSQLSession.ts
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,8 @@ export type ExecuteStatementOptions = {
* `false`/unset (default) runs the blocking direct-results path (faster,
* cancellable mid-compute); `true` submits and polls (returns a pending
* handle before completion). Default is sync, matching the python
* connector's `cursor.execute()`.
* connector's `cursor.execute()`. With `stagingAllowedLocalPath` configured,
* execution waits for staging completion regardless of this option.
*/
runAsync?: boolean;
maxRows?: number | bigint | Int64 | null;
Expand Down
3 changes: 3 additions & 0 deletions lib/contracts/IOperationBackend.ts
Original file line number Diff line number Diff line change
Expand Up @@ -113,6 +113,9 @@ export default interface IOperationBackend {
*/
getCachedResultMetadata?(): ResultMetadata | undefined;

/** Complete configured staging transfers before returning the operation. */
completeStagingOperation?(allowedPaths: string[]): Promise<void>;

/** Cancel the operation. */
cancel(): Promise<Status>;

Expand Down
10 changes: 2 additions & 8 deletions lib/kernel/KernelAuth.ts
Original file line number Diff line number Diff line change
Expand Up @@ -523,14 +523,8 @@ export function buildKernelHttpOptions(options: ConnectionOptions): KernelHttpOp
headers.push({ name: 'User-Agent', value: buildUserAgentString(userAgentEntry) });

const http: KernelHttpOptions = { customHeaders: headers };
// Per-connection socket read timeout (ms). The public `socketTimeout`
// ConnectionOption maps onto the kernel napi `socketTimeoutMs`
// (kernel `HttpConfig::request_timeout` / reqwest `Client::timeout`).
// Only forward a POSITIVE value: `socketTimeout: 0` means "disabled / wait
// indefinitely" on the Thrift path, but forwarding `0` would make reqwest
// time out immediately, so we omit it and let the kernel keep its (large)
// default — preserving the "effectively no idle timeout" semantics.
if (typeof socketTimeout === 'number' && socketTimeout > 0) {
// Preserve zero so native storage inactivity can be disabled like Thrift.
if (typeof socketTimeout === 'number' && socketTimeout >= 0) {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Medium — socketTimeout: 0 is now forwarded as socketTimeoutMs: 0. Bindings that still treat the value as the reqwest total timeout (the currently pinned ones) would time out immediately. The change has no unit test. Gate it on the coordinated binding release, add tests for 0 and positive values, and add a changelog note.

🔀 Found by Claude Opus 5.5 · confirmed by the judge

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Low — socketTimeout 0 is now forwarded to the binding, and positive values also set the storage idle timeout. There is no unit test for buildKernelHttpOptions covering this, and the CHANGELOG has no entry for it. Add tests for 0 and positive values, and a CHANGELOG note that it needs the new native release.

🔀 Found by Claude Opus 5.5 · confirmed by the judge

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Medium — socketTimeout: 0 is now forwarded to the native layer. A pre-change binding would treat 0 as an immediate reqwest timeout, and no test covers 0 or positive values. Add buildKernelHttpOptions tests and tie this change to the coordinated native release.

🔀 Found by Claude Opus 5.5 · confirmed by the judge

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Medium — socketTimeout: 0 is now forwarded as socketTimeoutMs: 0. The old comment said that value made reqwest time out immediately, and only the new native core treats it as disabling storage inactivity. With the 1.4.0 pins it can break requests, so omit zero until the compatible binding is pinned, or gate on binding capability.

🔀 Found by Claude Opus 5.5 · confirmed by the judge

http.socketTimeoutMs = socketTimeout;
}
return http;
Expand Down
Loading
Loading