Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@

## Unreleased

- Kernel backend: a custom `port` is honored over HTTPS, as on Thrift. It was dropped unless `https: false` was set. A non-default port folds into `host` (bare IPv6 hosts are bracketed); `port: 443`, a falsy `port` (`0`, `NaN`, `''`, `null`, `false`) and an omitted port leave `host` unchanged; a numeric string is accepted; a port embedded in `host` wins over a different `port` (a warning is logged); an out-of-range port throws `HiveDriverError`; a `host` with a path, query or fragment throws when the `port` is folded into it (`https: false`, or a non-443 `port` over HTTPS). With OAuth and a non-default port, the kernel reaches its OAuth endpoints at the same `host:port`, while Thrift keeps OAuth on the bare host (documented, not guarded).
- Reduce Arrow row materialization overhead on both backends by reading vectors directly and resolving scalar conversions once per batch. Object and array row values are unchanged.
- The Thrift token federation exchange (`enableTokenFederation`) now goes through the connection's agent, so `proxy`, `checkServerCertificate`, `customCaCert` and `clientCert`/`clientKey` apply to it. Previously it connected directly with Node's default agent, so behind a required proxy the exchange failed and the driver fell back to the original token. (#611)
- Kernel backend: a failed status poll while waiting for a statement (a `runAsync: true` statement, or a default execution that outlives the server's inline wait) and a failed result fetch in `hasMoreRows()` now throw the mapped driver error, such as `HiveDriverError: HTTP 500: Internal Server Error` once retries on HTTP 5xx run out, instead of a raw `Error` whose message is the internal `__databricks_error__:{...}` envelope. As in `fetchChunk()`, a failed fetch in `hasMoreRows()` also closes the statement. (#608)
Expand Down
18 changes: 9 additions & 9 deletions CONNECTION_PARAMETERS.md
Original file line number Diff line number Diff line change
Expand Up @@ -38,15 +38,15 @@ backend (the default), the **SEA / Kernel** backend, or both.

## Connection identity and backend selection

| Option | Type | Thrift | Kernel | Default Value | Note |
| ----------------------- | --------- | :----: | :----: | ------------- | -------------------------------------------------------------------------------------- |
| `host` | `string` | ✅ | ✅ | — (required) | Required on both. |
| `path` | `string` | ✅ | ✅ | — (required) | Kernel also derives `x-databricks-org-id` from a `?o=<id>` query parameter. |
| `port` | `number` | ✅ | ⚠️ | `443` / — | Kernel uses it only with `https: false`; otherwise put a non-default port in `host`. |
| `https` | `boolean` | ✅ | ✅ | `true` | `false` selects plain HTTP; see [Plain HTTP transport](#plain-http-transport). |
| `userAgentEntry` | `string` | ✅ | ✅ | — | Folded into the connector `User-Agent` on both. |
| `useKernel` (internal) | `boolean` | — | — | `false` | Selects the backend: `false`/omitted → Thrift, `true` → Kernel. |
| `clientId` (deprecated) | `string` | ✅ | ✅ | — | Runtime-only alias for `userAgentEntry`. `userAgentEntry` wins when both are supplied. |
| Option | Type | Thrift | Kernel | Default Value | Note |
| ----------------------- | --------- | :----: | :----: | ------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `host` | `string` | ✅ | ✅ | — (required) | Required on both. |
| `path` | `string` | ✅ | ✅ | — (required) | Kernel also derives `x-databricks-org-id` from a `?o=<id>` query parameter. |
| `port` | `number` | ✅ | ✅ | `443` / — | Kernel folds a non-default port into a bare `host` over HTTPS and with `https: false` (a bare IPv6 host is bracketed). A port already in `host` wins over `port`, with a warning when they differ; a falsy `port` means the default. With OAuth, the kernel uses the same `host:port` for its OAuth endpoints; Thrift keeps OAuth on the bare `host` (port 443). |
| `https` | `boolean` | ✅ | ✅ | `true` | `false` selects plain HTTP; see [Plain HTTP transport](#plain-http-transport). |
| `userAgentEntry` | `string` | ✅ | ✅ | — | Folded into the connector `User-Agent` on both. |
| `useKernel` (internal) | `boolean` | — | — | `false` | Selects the backend: `false`/omitted → Thrift, `true` → Kernel. |
| `clientId` (deprecated) | `string` | ✅ | ✅ | — | Runtime-only alias for `userAgentEntry`. `userAgentEntry` wins when both are supplied. |

Set `client.getConfig().useCloudFetch = false` (default `true`) before opening a session to fetch
complete results without cloud-storage downloads. On the kernel backend this
Expand Down
81 changes: 66 additions & 15 deletions lib/kernel/KernelAuth.ts
Original file line number Diff line number Diff line change
Expand Up @@ -830,23 +830,74 @@ export function buildKernelProxyOptions(options: ConnectionOptions): KernelProxy
return { proxy: out };
}

// The kernel prepends `https://` to a bare host but keeps an explicit scheme.
// It otherwise ignores `port`, so carry it into the URL unless `host` already has one.
function buildKernelHostName(options: ConnectionOptions): string {
if (options.https !== false) {
return options.host;
// `name:port` or `[ipv6]:port`; a bare IPv6 literal like `::1` has no port.
const EMBEDDED_PORT_RE = /^(\[[^\]]+\]|[^:]+):(\d+)$/;

// An explicit `port` is read as Thrift reads it (`options.port || 443`): a falsy value
// (unset, null, '', 0, NaN, false) means the default. A truthy value must be a whole
// port number, and a decimal string such as `'8443'` is accepted. Anything else throws,
// as Thrift also fails on it.
function explicitPort(port: unknown): number | undefined {
if (!port) return undefined;
let value = Number.NaN;
if (typeof port === 'number') value = port;
else if (typeof port === 'string' && /^[0-9]+$/.test(port)) value = Number(port);
if (value === 0) return undefined;
if (!Number.isInteger(value) || value < 1 || value > 65535) {
throw new HiveDriverError(`kernel backend: \`port\` must be an integer between 1 and 65535; got ${String(port)}.`);
}
return value;
}

// A bare IPv6 literal (colons, no brackets) needs brackets before a port is appended.
const bracketIpv6 = (authority: string): string =>
authority.includes(':') && !authority.startsWith('[') ? `[${authority}]` : authority;

// The kernel keeps a `host:port` authority but has no port setter, so an explicit port is
// folded into the host. Precedence, as on the pre-existing kernel path: a port embedded in
// `host` always wins. The explicit `port` applies only to a bare host, and when it differs
// from the embedded port the driver warns and connects to the embedded port. A trailing
// "/" is dropped before any port handling, as the kernel does.
function buildKernelHostName(options: ConnectionOptions, logger?: IDBSQLLogger): string {
const port = explicitPort(options.port);
const scheme = /^https?:\/\//i.exec(options.host)?.[0] ?? '';
const authority = options.host.slice(scheme.length).replace(/\/$/, '');
// A path, query or fragment in host cannot take a folded port: it is rejected when the port would
// be folded in (https: false, or HTTPS with a non-443 port). With no port, or with 443 over HTTPS,
// the host is passed through unchanged, as on origin/main (including `h.io/path:8443`).
if (port !== undefined && /[/?#]/.test(authority) && (options.https === false || port !== 443)) {
throw new HiveDriverError(
`kernel backend: host ${JSON.stringify(options.host)} has a path, query or fragment, so \`port\` ${port} ` +
'cannot be applied to it. Use a bare host or host:port.',
);
}
const match = EMBEDDED_PORT_RE.exec(authority);
const embedded = match ? Number(match[2]) : undefined;
if (embedded !== undefined && (embedded < 1 || embedded > 65535)) {
throw new HiveDriverError(`kernel backend: host carries port ${embedded}, which is outside 1..65535.`);
}
// 443 is the default that getConnectionOptions fills in, so it is not reported as a conflict.
if (port !== undefined && port !== 443 && embedded !== undefined && embedded !== port) {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Low — The warning skips port 443 because the comment says getConnectionOptions fills it in. But backend.connect(options) gets the raw caller options, so a 443 was set explicitly. {host:'h:8443', port:443} therefore drops the caller's port without a warning. Remove the port !== 443 exemption, or correct the comment.

🔀 Found by Claude Opus 5.5 · confirmed by the judge

logger?.log(
LogLevel.warn,
`kernel backend: host ${JSON.stringify(options.host)} carries port ${embedded} but \`port\` is ${port}. ` +
`Connecting to port ${embedded}; \`port\` is ignored.`,
);
}
// Fold a port into a bare host. `https: false` always folds a supplied port, while HTTPS
// folds only a non-default one (443 is the default and adds nothing).
if (embedded === undefined && port !== undefined && (options.https === false || port !== 443)) {
const hostPort = `${bracketIpv6(authority)}:${port}`;
return options.https === false ? `http://${hostPort}` : `${scheme}${hostPort}`;
}
// The kernel accepts a full URL in `host`, so drop an existing `http://` or
// `https://` scheme before applying HTTP (`https: false` selects the scheme).
// Then strip a trailing `/` like the Thrift URL builder does.
const host = options.host.replace(/^https?:\/\//i, '').replace(/\/$/, '');
// `name:port` or `[ipv6]:port`; a bare IPv6 literal like `::1` has no port.
const hasPort = /^(\[[^\]]+\]|[^:]+):\d+$/.test(host);
const port = options.port && !hasPort ? `:${options.port}` : '';
return `http://${host}${port}`;
if (options.https === false) return `http://${authority}`;
return `${scheme}${authority}`;
}

export function buildKernelConnectionOptions(options: ConnectionOptions): KernelNativeConnectionOptions {
export function buildKernelConnectionOptions(
options: ConnectionOptions,
logger?: IDBSQLLogger,
): KernelNativeConnectionOptions {
const { authType } = options as { authType?: string };

const base: {
Expand All @@ -859,7 +910,7 @@ export function buildKernelConnectionOptions(options: ConnectionOptions): Kernel
KernelHttpOptions &
KernelProxyOptions &
KernelFederationOptions = {
hostName: buildKernelHostName(options),
hostName: buildKernelHostName(options, logger),
httpPath: prependSlash(options.path),
// Match the NodeJS Thrift driver, which surfaces INTERVAL columns as
// strings. The kernel defaults to native Arrow interval/duration types;
Expand Down
2 changes: 1 addition & 1 deletion lib/kernel/KernelBackend.ts
Original file line number Diff line number Diff line change
Expand Up @@ -96,7 +96,7 @@ export default class KernelBackend implements IBackend {
// config (the same `ClientConfig` knobs the Thrift `HttpRetryPolicy` reads),
// converted from the connector's milliseconds to the kernel's whole seconds.
this.nativeOptions = {
...buildKernelConnectionOptions(options),
...buildKernelConnectionOptions(options, this.context.getLogger()),
...buildKernelRetryOptions(this.context.getConfig()),
...buildKernelTelemetryOptions(this.context.getConfig(), options, this.context.getLogger()),
};
Expand Down
34 changes: 34 additions & 0 deletions tests/unit/DBSQLClient.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -234,6 +234,40 @@ describe('DBSQLClient.connect', () => {
expect(connectionOptions.port).to.equal(3015);
});

it('keeps the Thrift URL for a host with an embedded port unchanged (current behavior, not changed by the kernel fix)', async () => {
const client = new DBSQLClient();

// Thrift appends options.port (443 by default) after the host, so an embedded port becomes
// `h:8443:443`, which node-fetch rejects as an invalid URL. This pins that current behavior.
const https = client['getConnectionOptions']({ ...connectOptions, host: 'h.databricks.com:8443', port: 443 });
expect(https.host).to.equal('h.databricks.com:8443');
expect(https.port).to.equal(443);
const httpsConnection = await new HttpConnection(https, client).getThriftConnection();
expect(httpsConnection.url).to.equal('https://h.databricks.com:8443:443/');

const http = client['getConnectionOptions']({
...connectOptions,
host: 'localhost:3015',
port: 80,
https: false,
});
const httpConnection = await new HttpConnection(http, client).getThriftConnection();
expect(httpConnection.url).to.equal('http://localhost:3015:80/');
});

it('treats a falsy port as the default (443 over HTTPS, 80 over HTTP), as the kernel path does', async () => {
const client = new DBSQLClient();

for (const port of [0, Number.NaN, undefined, null, '', false]) {
const label = String(port);
expect(client['getConnectionOptions']({ ...connectOptions, port: port as any }).port, label).to.equal(443);
expect(
client['getConnectionOptions']({ ...connectOptions, port: port as any, https: false }).port,
label,
).to.equal(80);
}
});

it('should only switch to plain HTTP for an explicit https: false', async () => {
const client = new DBSQLClient();

Expand Down
Loading
Loading