Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@

- Avoid intermediate decimal strings when materializing Arrow DECIMAL values with safe unscaled integers, on both backends.
- Reduce Arrow row materialization overhead on both backends by reading vectors directly and resolving scalar conversions once per batch. Object and array row values are unchanged.
- Kernel HTTPS traffic negotiates HTTP/2 with HTTP/1.1 fallback. Set `enableHttp2: false` to force HTTP/1.1. Configured proxies also force HTTP/1.1.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Medium — The changelog claims HTTP/2 negotiation and enableHttp2: false work, but the platform packages are pinned to 1.4.0, which has no such field in this diff. Add the published-package caveat and the kernel change reference, or land the pin bump with the entry.

🔀 Found by Claude Opus 5.5 · confirmed by the judge

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Duplicate of the release prerequisite at native/kernel/index.d.ts:781. Blocked on a kernel release that includes https://github.com/databricks/databricks-sql-kernel/pull/483; the dependency and generated-file updates listed in that thread remain required before merge.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Still open at b655703: CHANGELOG.md:6 still has no published-package caveat or kernel PR reference, and the pins are still not bumped.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This PR will merge only together with the KERNEL_REV and platform-pin bump, so the Unreleased entry will be accurate when it lands. A temporary caveat would be removed in that same push. The matching release of kernel #483 remains a merge prerequisite.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Still open at 0d563d5: platform pins remain 1.4.0 (package.json:95-102), and CHANGELOG.md:6 lacks the caveat or kernel #483 reference.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Medium — Every platform binding is still pinned to 1.4.0, which predates kernel #483. That means enableHttp2: false gets accepted and forwarded but probably has no effect in the shipped package, yet this entry says it works with no conditions. Either bump all the platform pins to a binding that has enableHttp2 in this PR, or say in the entry that it needs that release, reference kernel #483, and track the pin bump as follow-up.

🔀 Found by Claude Opus 5.5 · the judge couldn't confirm it

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Duplicate of the release prerequisite at native/kernel/index.d.ts:781. Blocked on a kernel release that includes https://github.com/databricks/databricks-sql-kernel/pull/483; the dependency and generated-file updates listed in that thread remain required before merge.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Still open at b655703: all platform pins remain on 1.4.0, and CHANGELOG.md:6 still describes enableHttp2 with no condition. Blocked on a kernel #483 release.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This PR will merge only together with the KERNEL_REV and platform-pin bump, so the Unreleased entry will be accurate when it lands. A temporary caveat would be removed in that same push. The matching release of kernel #483 remains a merge prerequisite.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Still open at 0d563d5: package.json:95-102 pins stay at 1.4.0. Keeping this open until the kernel #483 release and pin bump land.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Medium — The changelog says HTTP/2 is negotiated, but package.json still pins every kernel binding to 1.4.0. If 1.4.0 predates the kernel change, the option is silently ignored. Bump the pins in this PR, or state the required kernel package version and reference the kernel change.

🔀 Found by Claude Opus 5.5 · confirmed by the judge

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Duplicate of the release prerequisite at native/kernel/index.d.ts:781. Blocked on a kernel release that includes https://github.com/databricks/databricks-sql-kernel/pull/483; the dependency and generated-file updates listed in that thread remain required before merge.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Still open at b655703: the kernel pins are unchanged, and the CHANGELOG.md:6 entry doesn't name the required kernel release or link databricks-sql-kernel#483.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This PR will merge only together with the KERNEL_REV and platform-pin bump, so the Unreleased entry will be accurate when it lands. A temporary caveat would be removed in that same push. The matching release of kernel #483 remains a merge prerequisite.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Still open at 0d563d5: package.json:95-102 still pins kernel 1.4.0. Keep this open as a merge blocker until the bump lands.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Low — The changelog entry presents HTTP/2 negotiation as shipped, but it depends on a native package release that isn't pinned yet, and the entry doesn't mention that. State the required kernel release or package bump, or land the entry together with the pin update.

🔀 Found by Claude Opus 5.5 · confirmed by the judge

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Duplicate of the release prerequisite at native/kernel/index.d.ts:781. Blocked on a kernel release that includes https://github.com/databricks/databricks-sql-kernel/pull/483; the dependency and generated-file updates listed in that thread remain required before merge.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Still open at b655703: CHANGELOG.md:6 still doesn't name the required kernel release (databricks-sql-kernel#483) or the package bump.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This PR will merge only together with the KERNEL_REV and platform-pin bump, so the Unreleased entry will be accurate when it lands. A temporary caveat would be removed in that same push. The matching release of kernel #483 remains a merge prerequisite.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Still open at 0d563d5: KERNEL_REV and platform pins aren't bumped yet, so CHANGELOG.md:6 still describes support that isn't shipped. This resolves once the pin lands.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Low — The changelog entry presents HTTP/2 control as shipped, but the pinned 1.4.0 native packages don't support the field. State the required kernel release and native package bump, or land the pin bump with this entry.

🔀 Found by Claude Opus 5.5 · confirmed by the judge

- The Thrift token federation exchange (`enableTokenFederation`) now goes through the connection's agent, so `proxy`, `checkServerCertificate`, `customCaCert` and `clientCert`/`clientKey` apply to it. Previously it connected directly with Node's default agent, so behind a required proxy the exchange failed and the driver fell back to the original token. (#611)
- Kernel backend: a failed status poll while waiting for a statement (a `runAsync: true` statement, or a default execution that outlives the server's inline wait) and a failed result fetch in `hasMoreRows()` now throw the mapped driver error, such as `HiveDriverError: HTTP 500: Internal Server Error` once retries on HTTP 5xx run out, instead of a raw `Error` whose message is the internal `__databricks_error__:{...}` envelope. As in `fetchChunk()`, a failed fetch in `hasMoreRows()` also closes the statement. (#608)
- Honor `cloudFetchConcurrentDownloads` when opening kernel sessions, with a default window of 16 and explicit chunk-setting overrides. Values other than positive safe integers raise `HiveDriverError` at session open.
Expand Down
1 change: 1 addition & 0 deletions CONNECTION_PARAMETERS.md
Original file line number Diff line number Diff line change
Expand Up @@ -79,6 +79,7 @@ Per-statement `useCloudFetch` overrides remain unsupported on the kernel backend
| Option | Type | Thrift | Kernel | Default Value | Note |
| --------------------------- | ------------------------ | :----: | :----: | ------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `socketTimeout` | `number` (ms) | ✅ | ⚠️ | `900000` / `120000` | Kernel forwards only positive values. `0` means no timeout on Thrift but is omitted on Kernel, which then keeps its 120-second default. |
| `enableHttp2` | `boolean` | ❌ | ✅ | `true` | Negotiate HTTP/2 for session HTTPS traffic, with HTTP/1.1 fallback. `false` or an explicit/environment proxy forces HTTP/1.1, including bypassed hosts. |
| `proxy` | `ProxyOptions` | ✅ | ⚠️ | — | `{protocol, host, port, auth}`. Kernel supports HTTP/HTTPS proxies only; Thrift also supports the declared SOCKS variants. |
| `noProxy` (internal) | `string` | ❌ | ⚠️ | — | Runtime-only Kernel bypass list, forwarded as `bypassHosts` only when an explicit `proxy` is also supplied. |
| `customHeaders` | `Record<string, string>` | ✅ | ✅ | — | Sent on workspace, telemetry and feature-flag requests, not OAuth. Workspace calls ignore caller `Authorization`, `User-Agent`, `Content-Type`, `Content-Length`, `Transfer-Encoding`, `Connection`; Kernel also ignores `x-databricks-org-id`. |
Expand Down
7 changes: 7 additions & 0 deletions lib/contracts/IDBSQLClient.ts
Original file line number Diff line number Diff line change
Expand Up @@ -198,6 +198,13 @@ export type ConnectionOptions = {
*/
enableGeoSpatialSupport?: boolean;

/**
* Negotiate HTTP/2 for session HTTPS traffic, with HTTP/1.1 fallback.
* Defaults to true. False or a configured proxy forces HTTP/1.1,
* including bypassed destinations. Applies only with useKernel: true.
*/
enableHttp2?: boolean;

/**
* Extra HTTP headers sent on every request to the workspace (session,
* statement and metadata calls, on both backends) and on driver-owned
Expand Down
7 changes: 7 additions & 0 deletions lib/kernel/KernelAuth.ts
Original file line number Diff line number Diff line change
Expand Up @@ -238,6 +238,7 @@ export interface KernelTlsOptions {
* the kernel folds last-wins).
*/
export interface KernelHttpOptions {
enableHttp2?: boolean;
customHeaders?: Array<{ name: string; value: string }>;
socketTimeoutMs?: number;
}
Expand Down Expand Up @@ -523,6 +524,12 @@ export function buildKernelHttpOptions(options: ConnectionOptions): KernelHttpOp
headers.push({ name: 'User-Agent', value: buildUserAgentString(userAgentEntry) });

const http: KernelHttpOptions = { customHeaders: headers };
if (options.enableHttp2 !== undefined) {
if (typeof options.enableHttp2 !== 'boolean') {
throw new HiveDriverError('kernel backend: `enableHttp2` must be a boolean.');
}
http.enableHttp2 = options.enableHttp2;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟠 High — enableHttp2 is forwarded to the native binding, but every optionalDependency is still pinned to 1.4.0 and this diff doesn't bump them. An older binding drops the unknown field silently, so enableHttp2: false does nothing. Bump all platform pins to a binding that has the field, or add a capability check.

🔀 Found by Claude Opus 5.5 · confirmed by the judge

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Duplicate of the release prerequisite at native/kernel/index.d.ts:781. Blocked on a kernel release that includes https://github.com/databricks/databricks-sql-kernel/pull/483; the dependency and generated-file updates listed in that thread remain required before merge.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Still open at b655703: package.json:95-102 pins all platform bindings to 1.4.0; keeping this open until the kernel #483 release bump lands.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟠 High — enableHttp2 is forwarded to the native binding, but package.json still pins all eight @databricks/databricks-sql-kernel-* packages at 1.4.0, which the PR says lack the field. The option is silently ignored for npm consumers. Bump the platform pins and lockfile to the release that supports it, or have the router check for that support, before advertising the option.

🔀 Found by GPT-6.1 Sol · confirmed by the judge

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Duplicate of the release prerequisite at native/kernel/index.d.ts:781. Blocked on a kernel release that includes https://github.com/databricks/databricks-sql-kernel/pull/483; the dependency and generated-file updates listed in that thread remain required before merge.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Still open at b655703: the platform package pins and lockfile still need bumping to a kernel release that supports enableHttp2 before merge.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟠 High — The option is forwarded to native packages still pinned at 1.4.0 in package.json. The diff doesn't show those binaries supporting it, so the opt-out would be silently ignored for published installs. Bump KERNEL_REV and all eight platform pins, regenerate the binding, and verify the option, or gate the merge on the kernel release.

🔀 Found by both reviewers · confirmed by the judge

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Duplicate of the release prerequisite at #626 (comment). This PR is blocked on the kernel #483 release; KERNEL_REV, all eight platform pins and the lockfile must be updated, and native/kernel/index.* regenerated with npm run build:native, before merge.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Still open at 0d563d5: the 1.4.0 native pins remain, so the published binaries still ignore the option until the kernel #483 release ships and is bumped.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟠 High — The driver forwards enableHttp2 to native kernel packages that are all still pinned to 1.4.0 in package.json. Per the reviewers, that release has no such field, so enableHttp2: false would likely be silently dropped. Bump KERNEL_REV, the eight platform pins and the lockfile to the kernel release with the option, then regenerate the bindings. Alternatively, land this change together with that bump.

🔀 Found by both reviewers · confirmed by the judge

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Duplicate of the release prerequisite at #626 (comment). This PR is blocked on the kernel #483 release; KERNEL_REV, all eight platform pins and the lockfile must be updated, and native/kernel/index.* regenerated with npm run build:native, before merge.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Still open at 0d563d5: the kernel pins, KERNEL_REV, lockfile and regenerated bindings must ship with this change before merge.

}
// Per-connection socket read timeout (ms). The public `socketTimeout`
// ConnectionOption maps onto the kernel napi `socketTimeoutMs`
// (kernel `HttpConfig::request_timeout` / reqwest `Client::timeout`).
Expand Down
2 changes: 2 additions & 0 deletions native/kernel/index.d.ts

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

17 changes: 17 additions & 0 deletions tests/unit/kernel/connectionOptions.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,23 @@ import HiveDriverError from '../../../lib/errors/HiveDriverError';

const PAT = { host: 'h.databricks.com', path: '/sql/1.0/warehouses/abc', token: 'dapi-x' };

describe('Kernel HTTP/2 configuration', () => {
it('leaves the protocol default to the kernel', () => {
expect(buildKernelConnectionOptions(PAT)).not.to.have.property('enableHttp2');
});

for (const enableHttp2 of [true, false]) {
it(`preserves enableHttp2=${enableHttp2}`, () => {
expect(buildKernelConnectionOptions({ ...PAT, enableHttp2 }).enableHttp2).to.equal(enableHttp2);
});
}

it('rejects a non-boolean enableHttp2', () => {
const options = { ...PAT, enableHttp2: 'false' } as unknown as ConnectionOptions;
expect(() => buildKernelConnectionOptions(options)).to.throw(HiveDriverError, 'enableHttp2');
});
});

// Cast helper: the kernel connection-tuning/TLS options live on the internal
// surface, so tests build untyped option literals.
const opts = (extra: Record<string, unknown>) => ({ ...PAT, ...extra } as unknown as ConnectionOptions);
Expand Down
Loading