Skip to content

feat(kernel): honor WithPort on the kernel backend - #581

Open
cathleeny wants to merge 8 commits into
mainfrom
cathleeny/feat/kernel-honor-custom-port
Open

cathleeny wants to merge 8 commits into
mainfrom
cathleeny/feat/kernel-honor-custom-port

Conversation

@cathleeny

@cathleeny cathleeny commented Nov 19, 2026 •

Copy link
Copy Markdown
Collaborator

What type of PR is this?

  • Refactor
  • Feature
  • Bug Fix
  • Other

Description

On the kernel backend, WithPort (and the DSN port) is now honored. The kernel has no port setter, but it keeps a host:port authority in its host, so the connector folds a non-default port into the host it connects to.

This revision parses the host with Go's standard net/url instead of net.SplitHostPort plus hand-written bracket and path scans. The host is parsed once per call as the https authority the kernel connects to. The parse-once wrappers are gone, and buildKernelConfig, resolveKernelProxy and the tests call the same functions as before.

IPv6 literals must be bracketed. [::1] works, and [::1]:8443 works. A bare ::1 is rejected with ErrInvalidKernelConfig. It never connected on main either, because the kernel prepends https:// and its URL parse rejects an unbracketed literal.

Port and host rules:

  1. A port embedded in the host wins over a differing WithPort or DSN port. The connector logs a warning naming both ports and connects to the embedded port. It does not return an error. An explicit 443 is the default and is never warned about.
  2. A zero (unset) port means the default, and no :443 is added. With the default port (0 or 443) the host passes through unchanged.
  3. A trailing / is trimmed first, matching the kernel's normalise_host. WithServerHostname("https://h/") + WithPort(8443) connects to h:8443.
  4. An out-of-range WithPort (outside 1..65535) is rejected. An embedded port that is empty, non-numeric, or outside 1..65535 is rejected.
  5. With a non-default port, a host with a path, query or fragment is rejected, including an empty h# or h?. A host with user info (user@h) is rejected, and so are unbalanced brackets.
  6. With the default port, a path, query or fragment passes through unchanged, as before.

Everything else is unchanged: the non-https protocol rejection, WithTimeout, WithTransport, the other kernel rejections, and the Thrift path. Errors still wrap the same ErrInvalidKernelConfig sentinel.

Differences from the reviewed revision (52bd5ac)

  • A bare IPv6 literal (::1) is rejected at any port. Before, it was accepted at the default port and folded to [::1]:8443 with a custom port.
  • A host with a password (user:pw@h) is rejected at any port. Before, it was rejected too, because its SplitHostPort parse failed.
  • A host that net/url cannot parse (for example h%zz or h i) is rejected even at the default port, with ErrInvalidKernelConfig. Before, it passed through unchanged, and the kernel's WHATWG host parse rejects it, so it never connected. The failure moves earlier and gets a clearer error; no working connection is lost. A bracketed IPv6 host is unaffected.
  • Unchanged from 52bd5ac: a stray ] in a plain host name is still rejected. url.Parse accepts it, so the bracket check stays explicit.

Net change in the kernel files versus 52bd5ac: kernel_config.go +68 / -61, kernel_config_test.go +30 / -2, kernel_proxy_test.go +3 / -3, and one-line edits to the CHANGELOG, CONNECTION_PARAMETERS.md and README.md. kernel_proxy.go is unchanged from 52bd5ac.

Documented, not guarded: with OAuth (M2M or U2M) and a non-default port, the kernel reaches the OAuth endpoints at the same host:port as the connection. Thrift keeps OAuth on the bare host. Noted in CONNECTION_PARAMETERS.md, README.md and the CHANGELOG.

How is this tested?

  • Unit tests

  • E2E Tests

  • Manually

  • N/A

  • TestKernelCustomPort is table-driven. It covers the rules above, the bare-IPv6 rejection, and the h#, h?, user@h and user:pw@h cases, with the default and a custom port.

  • TestKernelCustomPort also covers the WithServerHostname("https://h/") + WithPort(8443) fold, and an embedded port with a trailing /. The proxy endpoint test covers a bracketed IPv6 literal with a port.

  • CGO_ENABLED=0 go test -count=1 ./... passes. go vet, gofmt -s and the repo's golangci-lint config (v2.15.0, default build) report no issues.

  • Kernel-tagged suite (-tags databricks_kernel, bindings v1.4.0): every package passes except TestKernelCloudFetchDisabledPreservesUnsupportedServerError. That test fails the same way at 52bd5ac, so it is not caused by this change.

  • Live, kernel and Thrift paths, DBSQL and Reyden warehouses, at this commit. See the Live section below.

Live

Probe built from this commit. Each case runs one connect plus SELECT 1, and the effective backend is checked. Warm samples are in milliseconds.

Case DBSQL Reyden
Kernel WithPort(443) pass, *kernel.KernelBackend, warm 570/263/235 pass, warm 99/91/92 on retry
Kernel embedded :443, WithPort(0) pass, warm 205/199/200 pass on retry, warm 217/89/113
Kernel embedded :443, WithPort(8443) (embedded wins, warning logged) pass, warm 193/196/205 pass, warm 206/90/94
Thrift WithPort(443) pass, *thrift.Backend, warm 177/175/169 not run (the Reyden Thrift path is not supported; it reroutes to the kernel)

The first Reyden attempt of two cases hit the 100 s wrapper deadline during a slow spell, with token-federation fallback logged. Both passed on a single retry.

No 52bd5ac live run was made, so these are not compared against the reviewed revision. The warm numbers are for reference only.

Performance

Host/port resolution is called three times per connect (validation, the folded Host, and the proxy endpoint). Each call now parses with net/url. The table gives the median of 10 go test -bench runs per case, run one at a time on the same machine, for 52bd5ac and this commit. Allocations are per call.

Benchmark (median ns/op) Case 52bd5ac This commit
validateKernelHostPort bare host, port 8443 45.1 (1 alloc) 213.5 (2 allocs)
validateKernelHostPort host:port, embedded 43.0 (0 allocs) 248.9 (2 allocs)
validateKernelHostPort bracketed IPv6, port 8443 43.4 (1 alloc) 246.5 (3 allocs)
kernelHostWithPort bare host, port 8443 61.9 (3 allocs) 258.7 (4 allocs)
kernelHostWithPort host:port, embedded 2.2 (0 allocs) 233.4 (2 allocs)
kernelHostWithPort bracketed IPv6, port 8443 59.9 (3 allocs) 289.6 (5 allocs)
validate + fold bare host, port 8443 111.0 524.5
validate + fold host:port, embedded 46.1 944.2
validate + fold bracketed IPv6, port 8443 106.5 557.0

The net/url parse costs roughly 170 to 200 ns per call. On the connect path that is about 0.5 to 0.9 µs in total, against a connect that takes milliseconds or more. The 52bd5ac host:port fold was a 2 ns early return, and this commit parses even then to keep the rejection rules.

Related Tickets & Documents

Related PRs: databricks/databricks-sql-python#1071, databricks/databricks-sql-nodejs#636, https://github.com/databricks/databricks-driver-test/pull/2188

This pull request and its description were written by Isaac.

The kernel backend rejected any WithPort other than 443. The kernel has no port
setter, but it keeps a `host:port` authority in its host, so the connector now
folds a non-default port into the host it passes to the kernel.

- kernelHostWithPort adds a non-default port with net.JoinHostPort (IPv6-safe).
  The default 443 and an unset port add nothing.
- validateKernelHostPort rejects an out-of-range port, a host with unbalanced IPv6
  brackets (for example `[::1` or `h.databricks.com]`), and a supplied port that
  disagrees with a port embedded in the host. An unset port (WithDefaults fills
  443) defers to the embedded port. An explicit WithPort(443) or DSN port is
  supplied, so it must match an embedded port too (AC-P3: never drop a port).
- UserConfig gains an internal PortExplicit flag, set by WithPort and the DSN
  parser. It is not a public option: internal/ is not importable by users.
- Non-https protocols and the other kernel rejections are unchanged.

With OAuth (M2M and U2M) and a non-default port, the kernel reaches the OAuth
endpoints at the same host:port as the connection. Thrift keeps OAuth on the bare
host. This is documented in CONNECTION_PARAMETERS.md, the README and the
CHANGELOG, and is not guarded.

Tests: one table-driven TestKernelCustomPort covers the host mapping, the
validation (accept and reject, including explicit 443 and unbalanced brackets),
and the kernel Host. The config, connector and driver tests expect PortExplicit
where WithPort or the DSN set the port.

Co-authored-by: Isaac <no-reply@databricks.com>
Signed-off-by: Cathleen Yan <58714163+cathleeny@users.noreply.github.com>

@peco-review-bot peco-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔀 Cross-reviewed: two independent reviews (Claude Opus 5.5 · GPT-6.1 Sol), checked by a judge model (Claude Sonnet 5.5).

Verdict: 1 Medium

Looks good: one low concern. The port folding, the validation and the PortExplicit plumbing check out, including DeepCopy and the DSN path. Nit: the PortExplicit disposition is listed as "forwarded", but it is only consumed by validation.

This overview is Claude Opus 5.5's initial take; GPT-6.1 Sol added findings it doesn't mention — see the findings below.

Comment thread kernel_config.go Resolved
proxyForEndpointFunc appended the port to cfg.Host even when the host already carried one, so host:8443 with Port 8443 built an invalid endpoint, the proxy was dropped and HTTPS_PROXY was ignored. It now uses the same authority as kernelHostWithPort, which leaves an embedded port alone and brackets IPv6 literals.

Co-authored-by: Isaac <no-reply@databricks.com>
Signed-off-by: Cathleen Yan <58714163+cathleeny@users.noreply.github.com>

@peco-review-bot peco-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔀 Cross-reviewed: two independent reviews (Claude Opus 5.5 · GPT-6.1 Sol), checked by a judge model (Claude Sonnet 5.5).

Verdict: 1 Nit

Looks good, with 1 low concern. The port folding and validation are correct for IPv4, IPv6 and embedded-port cases, and PortExplicit is set by both entry points (WithPort and the DSN parser) and kept by DeepCopy. Nit: the out-of-range error message says "want 1..65535", but port 0 is accepted as unset.

Other findings

  • ⚪ Nit — The Reyden auto-recovery path (openSessionWithReydenFallback → getKernelBackend → buildKernelConfig) now folds a custom port into the kernel Host, where it was rejected before. That is reasonable, but no test or CHANGELOG line covers the recovery path. Add a recovery test with WithPort(8443) that asserts the kernel Host, and mention recovery in the CHANGELOG entry. (Found by Claude Opus 5.5 · confirmed by the judge)

Address the review nits on the kernel custom-port change:

- The out-of-range WithPort error now says 0 is accepted as the default.
- PortExplicit is classified as "validated", not "forwarded": it is consumed
  by validateKernelHostPort and never sent to the kernel.
- A Reyden recovery test with WithPort(8443) asserts that the kernel Host is
  host:8443 after the Thrift KP001 rejection. The CHANGELOG entry mentions the
  recovery path.

Co-authored-by: Isaac <no-reply@databricks.com>
Signed-off-by: Cathleen Yan <58714163+cathleeny@users.noreply.github.com>
@cathleeny

Copy link
Copy Markdown
Collaborator Author

Fixed the review nits in 263c741:

  • Port range message (review 5980950905): the out-of-range WithPort error now reads "want 1..65535, or 0 for the default", which matches the accepted unset value.
  • Reyden recovery coverage (review 5980950905): TestReydenReactiveRecovery has a new case. A Thrift KP001 rejection with WithPort(8443) recovers onto the kernel, and the case asserts that the kernel Host is reyden.example.com:8443 and that the port passes validation. The CHANGELOG entry now mentions the recovery path.
  • PortExplicit disposition (review 5980504145): the field is now classified as "validated", not "forwarded". This is the same label the experimental classification uses for TLSClientCertConfigured. The flag is consumed by validateKernelHostPort and never sent to the kernel.

CGO_ENABLED=0 go test -count=1 ./... passes for all packages, go vet ./... is clean, and gofmt is clean.

Automated reply from the driver feature factory (Isaac). Not reviewed by a human.

@peco-review-bot peco-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔀 Cross-reviewed: two independent reviews (Claude Opus 5.5 · GPT-6.1 Sol), checked by a judge model (Claude Sonnet 5.5).

✅ No issues identified.

Comment thread kernel_config.go Resolved
cathleeny and others added 5 commits November 19, 2026 23:13
…hPort

Fold a trailing "/" in the host before any port handling, matching the
kernel's normalise_host, so WithServerHostname("https://h/") plus
WithPort(8443) connects to h:8443 instead of h/:8443, and
"https://h:8443/" keeps working.

Port precedence now matches the Node and Python kernel paths:
- A port embedded in the host wins over a differing WithPort. The driver
  logs a warning naming both ports and connects to the embedded port.
- A zero (unset) WithPort means the default, with no :443 added.
- A host with a path, query or fragment after the trimmed "/" is
  rejected with ErrInvalidKernelConfig. Base also rejected it: the kernel
  builds {host}/api/2.0/sql, so h/x sent requests to /x/api/2.0/sql.
- Out-of-range WithPort and out-of-range or malformed embedded ports stay
  rejected. Thrift also fails on them: an embedded port plus the default
  443 yields an unparseable https://h:8443:443 URL, -1 is an invalid port,
  and 70000 fails to dial.

Base behavior (origin/main): validateKernelConfigContext rejected any
WithPort other than 0 or 443 with ErrNotSupportedByKernel, and passed the
host through unchanged, so an embedded port connected to that port.

Tests: table cases for h/, h:8443/, h/x, [::1]/, embedded-vs-WithPort
(warn, embedded wins), explicit 443 against an embedded port, and an
invalid port (still rejected). Proxy endpoint test for a trailing-slash
host. Kernel loopback probe: embedded port dialed, explicit port never.

Co-authored-by: Isaac <no-reply@databricks.com>
Signed-off-by: Cathleen Yan <58714163+cathleeny@users.noreply.github.com>
…xplicit 443

- A host with a path or query is rejected before the embedded-port parse, so
  "h.io/path:8443" with an explicit port cannot slip through. The old code passed
  the host verbatim to the kernel, which builds {host}/api/2.0/sql, so these
  configs never connected.
- The differing-port warning is skipped for an explicit 443 (the default).
- Thrift is unchanged. TestThriftEndpointUnchanged pins its current endpoint
  strings for the same port inputs, including the embedded-port-plus-443 case.

Co-authored-by: Isaac <no-reply@databricks.com>
Signed-off-by: Cathleen Yan <58714163+cathleeny@users.noreply.github.com>
- A host with a path, query or fragment is now rejected only when a
  non-default WithPort would have to be folded into it. The check still runs
  after the trailing "/" trim and before the embedded-port parse. With no port
  or 443 the host goes to the kernel unchanged, as on the base branch, where
  the kernel keeps a path prefix ({host}/api/2.0/sql). The base branch rejected
  every non-default WithPort, so nothing that worked before is rejected now.
- Remove the internal PortExplicit field. Once an explicit 443 stopped
  warning, nothing read it, so WithPort, ParseDSN, DeepCopy and their tests
  are back to the base branch.

Co-authored-by: Isaac <no-reply@databricks.com>
Signed-off-by: Cathleen Yan <58714163+cathleeny@users.noreply.github.com>
Resolve the CHANGELOG.md conflict by keeping both Unreleased entries: this PR's
kernel port entry and main's non-finite float rendering entries. No code conflicts.

Co-authored-by: Isaac <no-reply@databricks.com>
Signed-off-by: Cathleen Yan <58714163+cathleeny@users.noreply.github.com>
The kernel path folds WithPort into cfg.Host. The host is now parsed once per
call with net/url, as the https authority the kernel connects to, instead of
net.SplitHostPort plus hand-written bracket and path scans.

Rules (unchanged unless noted):
- A trailing "/" is trimmed first. With the default port (0 or 443) the host
  passes through unchanged.
- A port embedded in the host wins over a differing WithPort, with a warning.
  An explicit 443, or 0, is the default and is never warned about.
- WithPort outside 1..65535 is rejected. An embedded port that is empty,
  non-numeric or outside 1..65535 is rejected.
- With a non-default port, a host with a path, query, fragment (including an
  empty "h#" or "h?") or user info is rejected. Unbalanced brackets are rejected.
- Errors wrap ErrInvalidKernelConfig as before.

Behavior changes from the reviewed revision (52bd5ac):
- An IPv6 literal must be bracketed. A bare "::1" is rejected at any port
  (net/url cannot parse it). It never connected on main either: the kernel
  prepends https:// and its parse rejects it.
- A host with a password ("user:pw@h") is rejected at any port.
- A host net/url cannot parse (for example "h%zz" or a space) is rejected
  even at the default port, where it used to pass through to the kernel.

The parse-once wrappers and the separate warning helper are gone:
validateKernelHostPort logs the embedded-port warning, and kernelHostWithPort
is the only fold, so buildKernelConfig and resolveKernelProxy keep their
reviewed signatures.

Co-authored-by: Isaac <no-reply@databricks.com>
Signed-off-by: Cathleen Yan <58714163+cathleeny@users.noreply.github.com>

@peco-review-bot peco-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔀 Cross-reviewed: two independent reviews (Claude Opus 5.5 · GPT-6.1 Sol), checked by a judge model (Claude Sonnet 5.5).

Verdict: 2 Medium · 1 Low

Mostly solid, with 1 medium and 1 low concern. The port folding and validation are well tested, and the Reyden recovery and production paths both go through newKernelBackend's validation. The main problem is that a password in the host can end up in error messages.

This overview is Claude Opus 5.5's initial take; GPT-6.1 Sol added findings it doesn't mention — see the findings below.

Comment thread kernel_config.go
}
h.name = u.Hostname()
h.port = u.Port()
h.hasPort = h.port != "" || strings.HasSuffix(u.Host, ":")

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Medium — net/url accepts https://::1 and reads the trailing 1 as a port, so a bare IPv6 host passes validation and WithPort is ignored. ParseDSN also strips brackets (ucfg.Host = parsedURL.Hostname()), so DSN IPv6 hosts reach this path unbracketed. Detect unbracketed IPv6 explicitly and re-bracket DSN-derived hosts.

🔀 Found by GPT-6.1 Sol · confirmed by the judge

Comment thread kernel_config.go
h.raw, port, dbsqlerr.ErrInvalidKernelConfig)
}
if h.password {
return fmt.Errorf("databricks: host %q carries a password: %w", h.raw, dbsqlerr.ErrInvalidKernelConfig)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Medium — The credential and parse-error messages format the raw host, which includes the userinfo and password, into caller-visible errors. Format only the host name, or strip the userinfo, before building these errors.

🔀 Found by both reviewers · confirmed by the judge

Comment thread kernel_config.go
return fmt.Errorf("databricks: host %q carries credentials, so WithPort %d cannot be applied: %w",
h.raw, port, dbsqlerr.ErrInvalidKernelConfig)
}
if h.password {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Low — A host with a password is now rejected at every port, but the CHANGELOG and README only describe rejecting credentials with a non-default port. Document the rejection at all ports, or drop it.

🔀 Found by Claude Opus 5.5 · confirmed by the judge

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants