Skip to content

feat(kernel): honor _port and _connection_uri on use_kernel connections - #1071

Open
cathleeny wants to merge 12 commits into
mainfrom
cathleeny/feat/kernel-honor-port-uri-parity
Open

cathleeny wants to merge 12 commits into
mainfrom
cathleeny/feat/kernel-honor-port-uri-parity

Conversation

@cathleeny

@cathleeny cathleeny commented Nov 19, 2026 •

Copy link
Copy Markdown
Collaborator

What type of PR is this?

  • Refactor
  • Feature
  • Bug Fix
  • Other

Description

Honors the Thrift-style _port and _connection_uri overrides on the kernel backend (use_kernel=True). The kernel takes one host and one http_path, so both are resolved connector-side in session.py (_kernel_host_and_path). No kernel change.

Host and port are parsed with the standard library: urllib.parse.urlsplit for host[:port] and int() for the port. There is no hand-written host regex. server_hostname passes through unchanged unless a port is folded in. IPv6 literals must be bracketed ([::1]).

Port precedence, the same in the Python, Node and Go drivers:

  • A port embedded in server_hostname always wins. _port applies only to a bare host. If a non-default _port differs from the embedded port, a WARNING names both ports, and the connection uses the embedded port. It does not raise. An explicit _port=443, the default, never warns.

  • A falsy _port (None, 0, '', False) means the default. No :443 is added.

  • A trailing / on server_hostname is trimmed before port handling.

  • A truthy invalid _port (non-numeric, or outside 1..65535) raises ValueError, because Thrift also fails on it (see Base behavior).

  • An explicit _port with a server_hostname that has a path, query or fragment raises ValueError. The check runs before the port is parsed, so h.io/path:8443 is rejected rather than read as port 8443. The kernel builds its URL from the host, so these configs never connected. A host without _port passes through unchanged, as on main.

  • In this driver, an explicit _port with a malformed server_hostname raises ValueError. Malformed means a trailing :, a broken or unbracketed IPv6 literal, a non-numeric or out-of-range embedded port, or user:password@ credentials. The error message redacts the host. A host with user info but no password (user@h) is still folded, as before.

  • _connection_uri (http(s)://host[:port]/path?query) sets host, port, path and query, and wins over _port, as on Thrift. Scheme-less, non-http(s), path-less and credential-bearing URIs raise ValueError. URI-only use (empty server_hostname and http_path) works.

  • Some auth types count as OAuth for this check:

    • U2M;
    • Azure service principal;
    • a JWT key file;
    • M2M client id and secret;
    • an access_token that is a JWT from an issuer other than server_hostname.

    With any of these, a _connection_uri whose origin (scheme, host, port) differs from https://server_hostname raises before a kernel session opens. PAT is unaffected, and so is a JWT whose issuer is server_hostname.

  • The SPOG x-databricks-org-id header follows the resolved path when _connection_uri changes it.

Thrift is unchanged. session.py passes server_hostname and _port to ThriftDatabricksClient verbatim, and thrift_backend.py applies port or 443. A unit test pins this for the same configs.

CONNECTION_PARAMETERS.md now lists _port and _connection_uri as Kernel options, with the rules above.

Base behavior (origin/main)

  • session.py passes server_hostname and http_path to KernelDatabricksClient verbatim. The kernel path ignores _port entirely. An embedded host port passes through, so the kernel connects to it, which is why the embedded port wins.
  • client.py passes port=kwargs.get("_port", 443) only to Thrift. There is no _connection_uri on main.
  • Thrift (thrift_backend.py lines 170-176): port = port or 443, then "{host}:{port}/{path}". A falsy _port therefore means 443. A non-numeric, out-of-range, negative or boolean _port builds a URL that urllib3 rejects with LocationParseError (checked with the connector's urllib3 for abc, 70000, -1, True, 8443.5).

Foreign-issuer JWT as access_token

Reviewer thread (session.py, _check_oauth_origin): the kernel wraps every auth provider in FederationProvider using the connection host, PAT included. A JWT from a foreign issuer passed as access_token would therefore be exchanged at the _connection_uri origin, while Thrift exchanges it at server_hostname. The guard now treats such a token as OAuth. It reuses the connector's decode_token and is_same_host helpers, the same ones the Thrift token-federation path uses. A mismatched origin raises before any request.

Differences from the previous revision of this PR

  • A differing embedded and explicit port used to raise (Conflicting ports). It now connects to the embedded port with a warning, and _port=443 no longer warns at all.
  • An empty _port ('') and False used to raise. They now mean the default.
  • An explicit _port with a host that has a path, query or fragment used to pass through verbatim, and the kernel then built a URL with the path in the authority. It now raises.
  • "0" (a string) as _port is treated as the default. Thrift would dial port 0. This is a relaxation.
  • Host parsing moved from a regex to urlsplit. The behavior tables are unchanged. Two side effects:
    • The OAuth origin check now compares the parsed host name. A server_hostname such as h.io/x or user@h with a matching _connection_uri origin now passes where it was refused.
    • h:70000, h:abc and [::1 now share one malformed-or-out-of-range message.
  • A foreign-issuer JWT access_token is now subject to the OAuth origin check (above).
  • A bare IPv6 host with _port used to be bracketed and folded (::1 became [::1]:8443). It now raises; bracket it instead. Bare IPv6 never worked on the kernel path on main, because the kernel's URL parser rejects it. Without _port, the host still passes through unchanged.

Overlap with databricks-sql-python#915

#915 is another engineer's open PR. It changes the same function (_kernel_host_and_path) for the same feature. This branch is a self-contained change on main (b099dd6), with no commits from #915. Only one of the two should merge.

This PR also aligns validation with Thrift. It rejects inputs Thrift rejects: non-numeric or out-of-range ports, and path-less or scheme-less URIs. #915 defaulted scheme-less URIs to https and kept the original path. Coordinating with #915's owner is a human step.

How is this tested?

  • Unit tests

  • Other

  • Unit (tests/unit, -m "not realkernel"): every test file run in its own process passes (test_init_file.py needs the repo as cwd and passes from there). Running test_session.py and test_kernel_client.py in one process fails the real-wheel tests, because the latter installs a stub kernel module.

  • tests/unit/test_session.py: 196 pass. The table cases cover:

    • the resolver, accept and reject, including the path/query/fragment, user:password@ and trailing-: rejections with _port, the urlsplit edge cases (case, user info, broken IPv6, trailing /), and the scheme cases;
    • the embedded-wins warning (caplog) and the 443 no-warning case;
    • the OAuth origin guard, including foreign-issuer and same-issuer JWT access_token cases;
    • the Thrift pin (test_thrift_receives_host_and_port_unchanged).
  • tests/unit/test_kernel_client.py: 282 pass.

  • Realkernel (-m realkernel, databricks-sql-kernel==1.4.0): 5 pass. The loopback tests are test_kernel_dials_over_tls (bare host with _port, and URI-only) and test_kernel_embedded_port_wins_over_explicit_port (server_hostname 127.0.0.1:P1 with _port=P2). The TCP listener on P1 receives the TLS handshake (first byte 0x16), and the listener on P2 receives no connection.

  • black --check src: clean.

  • Microbenchmark of the host resolver, median per call: within 1-3% of the regex version for bare hosts, host:port and bracketed IPv6.

  • Live (kernel path, DBSQL 417ec0c098c5d0b6 and Reyden 0000000000e01f56, at f242c0f; the final commit only changes bare-IPv6 handling, which no live case uses): explicit _port 443, _connection_uri overriding _port=1, and URI-only all pass on both warehouses. Thrift passes on DBSQL. The Reyden Thrift rows show unexpected-transport, the tracked baseline exception.

  • Not covered live: a host with an embedded :443 (the probe has no such case).

This pull request and its description were written by Isaac.

The kernel backend (use_kernel=True) ignored the Thrift-style _port and
_connection_uri overrides. Resolve both connector-side onto the kernel's single
host and http_path, with the Thrift rules:

- _port (1..65535; 0 or None mean the default) folds into server_hostname as
  host:port. The default 443 adds nothing, IPv6 literals are bracketed, and an
  embedded port equal to _port is accepted. An empty server_hostname with _port
  raises ValueError.
- _connection_uri (http(s)://host[:port]/path?query) replaces host, port, path
  and query, and wins over _port. Scheme-less, non-http(s), path-less and
  credential-bearing URIs raise ValueError. URI-only use works.
- With OAuth (U2M, Azure service principal, JWT key file, or M2M client id and
  secret), a _connection_uri whose origin differs from https://server_hostname
  raises before a kernel session opens. PAT is not affected. A non-default _port
  with OAuth is not guarded: the token exchange goes to the same host:port, which
  is documented.
- The SPOG x-databricks-org-id header follows the resolved path.

Tests cover the resolver tables, the OAuth origin guard, the SPOG re-derivation,
and a real-wheel loopback dial proving the kernel reaches the requested port
over TLS for _port and for _connection_uri.

Co-authored-by: Isaac <no-reply@databricks.com>
Signed-off-by: Cathleen Yan <58714163+cathleeny@users.noreply.github.com>
The helper returned Optional[str] for the resolved path, but KernelDatabricksClient takes a str, so mypy fails in check-types on every supported Python. Session.__init__ already types http_path as str, so this is an annotation-only change with no runtime effect.

Co-authored-by: Isaac <no-reply@databricks.com>
Signed-off-by: Cathleen Yan <58714163+cathleeny@users.noreply.github.com>

@peco-review-bot peco-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔀 Cross-reviewed: two independent reviews (Claude Opus 5.5 · GPT-6.1 Sol), checked by a judge model (Claude Sonnet 5.5).

Verdict: 2 Low

Looks good: two low concerns. Resolver, OAuth origin guard and SPOG re-derivation match Thrift's precedence and have thorough table tests. Nit: a few new lines (the Conflicting-ports f-string, the _split_host and _uri_host_and_path docstrings) exceed the 100-char limit in CONTRIBUTING.md.

Comment thread src/databricks/sql/session.py Resolved
Comment thread src/databricks/sql/session.py Resolved

@peco-review-bot peco-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔀 Cross-reviewed: two independent reviews (Claude Opus 5.5 · GPT-6.1 Sol), checked by a judge model (Claude Sonnet 5.5).

Verdict: 2 Low

Looks good, with 1 low concern. The resolver, the OAuth origin guard and the SPOG re-derivation match the auth bridge's routing and are well covered by tests. Nit: a few new lines go past the 100-char limit (the Conflicting ports f-string and the _split_host/_uri_host_and_path docstrings), and the AC-P4 ticket reference in the docstring.

This overview is Claude Opus 5.5's initial take; GPT-6.1 Sol added findings it doesn't mention — see the findings below.

Comment thread src/databricks/sql/session.py Resolved
Comment thread src/databricks/sql/session.py Resolved
A kernel connection without _port or _connection_uri again passes server_hostname through unchanged, so odd hosts no longer raise on the default path. The OAuth origin guard now skips connections the kernel auth bridge routes to PAT: an access_token with no JWT or client-secret M2M credentials, even when auth_type is databricks-oauth. azure-sp-m2m and M2M/JWT stay guarded.

Co-authored-by: Isaac <no-reply@databricks.com>
Signed-off-by: Cathleen Yan <58714163+cathleeny@users.noreply.github.com>

@peco-review-bot peco-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔀 Cross-reviewed: two independent reviews (Claude Opus 5.5 · GPT-6.1 Sol), checked by a judge model (Claude Sonnet 5.5).

Verdict: 1 Low

Looks good, with 1 low concern. The resolver, the OAuth predicate (which matches the auth bridge's routing) and the SPOG re-derivation all check out. Nits: session.py:135 and the _split_host/_uri_host_and_path docstrings go past the 100-char limit, and test_session.py imports databricks.sql twice.

Comment thread src/databricks/sql/session.py Resolved
Wraps the origin-guard condition to the repo's black line length (check-linting). The guard no longer treats a custom credentials_provider as OAuth: the kernel auth bridge rejects that provider with NotSupportedError, so no OAuth origin is involved.

Co-authored-by: Isaac <no-reply@databricks.com>
Signed-off-by: Cathleen Yan <58714163+cathleeny@users.noreply.github.com>

@peco-review-bot peco-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔀 Cross-reviewed: two independent reviews (Claude Opus 5.5 · GPT-6.1 Sol), checked by a judge model (Claude Sonnet 5.5).

✅ No issues identified.

Comment thread CHANGELOG.md Resolved
# M2M credentials to PAT, even for auth_type='databricks-oauth'; azure-sp-m2m
# is resolved before that branch and stays OAuth. A custom credentials_provider
# is rejected by the bridge, so it is not an OAuth origin here.
routed_to_pat = (

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The kernel wraps every auth provider, PAT included, in FederationProvider using the connection host (session.rs:1394, host = the _connection_uri origin here). So a foreign-issuer JWT passed as access_token (Azure AD, BYOT, WIF) is POSTed to {_connection_uri host}/oidc/v1/token. Thrift does that exchange at server_hostname (auth.py:70) and the URI host only sees the minted token. _check_oauth_origin classifies every access_token as PAT, so this passes the guard with or without identity_federation_client_id. Suggest treating an access_token that decodes as a JWT with a non-server_hostname issuer as OAuth here (decode_token / is_same_host in auth_utils.py), plus a test.

A port embedded in server_hostname now wins over a differing _port, with a warning, as on main, where the kernel connected to the embedded port. A falsy _port (None, 0, '', False) means the default, as on Thrift (port or 443). A trailing '/' is trimmed before port handling. A truthy invalid _port still raises, as Thrift fails on it (LocationParseError).

Co-authored-by: Isaac <no-reply@databricks.com>
Signed-off-by: Cathleen Yan <58714163+cathleeny@users.noreply.github.com>

@peco-review-bot peco-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔀 Cross-reviewed: two independent reviews (Claude Opus 5.5 · GPT-6.1 Sol), checked by a judge model (Claude Sonnet 5.5).

Verdict: 1 Low

Looks good, no blocking issues. The OAuth origin guard's PAT-routing predicate matches the precedence in auth_bridge.kernel_auth_kwargs. Nits: an embedded port plus a path with _port set raises a misleading "malformed IPv6" error, and test_resolved_kernel_host_and_path_reach_kernel_client never closes its connections.

This overview is Claude Opus 5.5's initial take; GPT-6.1 Sol added findings it doesn't mention — see the findings below.

Comment thread src/databricks/sql/session.py Resolved
…ernel port rules

Check the host for a path, query or fragment before the port is parsed, so 'h.io/path:8443' with _port cannot read as port 8443. Only explicit _port is affected; a host without _port passes through unchanged, as on main. Add Thrift tests pinning that host and _port reach ThriftDatabricksClient unchanged. Update CONNECTION_PARAMETERS.md: _port and _connection_uri are now kernel options, with the embedded-port and falsy rules and the OAuth origin rule. Co-authored-by: Isaac <no-reply@databricks.com>

Signed-off-by: Cathleen Yan <58714163+cathleeny@users.noreply.github.com>

@peco-review-bot peco-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔀 Cross-reviewed: two independent reviews (Claude Opus 5.5 · GPT-6.1 Sol), checked by a judge model (Claude Sonnet 5.5).

✅ No issues identified.

cathleeny and others added 2 commits November 20, 2026 00:44
Replace the _HOSTPORT_RE regex, the _port regex and the manual bracketing in the kernel host folding with urllib.parse.urlsplit, int() and an ipaddress pre-check for bare IPv6 literals. The original host string still passes through unchanged unless a port is folded in; folded IPv6 hosts are bracketed.

_check_oauth_origin treats an access_token that decodes as a JWT whose issuer is not server_hostname as OAuth. The kernel wraps every provider with the connection host, so such a token would be exchanged at the _connection_uri origin, while Thrift exchanges it at server_hostname. A mismatched origin now raises before any request. PATs and same-host issuers are unaffected.

The port-conflict warning is skipped when _port is the default 443, matching Node. Non-default ports still warn on a mismatch.

Behavior-table tests in test_session.py pass unchanged. One warning test changed: test_embedded_port_wins_over_differing_explicit_port lost its ('h.io:8443', 443) row, which asserted the 443 warning. That case moved to test_embedded_port_with_default_443_does_not_warn under the new rule.

Known differences from bc7c09e: the OAuth origin check now compares the parsed hostname, so a server_hostname such as 'h.io/x' or 'user@h' with a matching _connection_uri origin passes where it was previously refused. The error text for 'h:70000', 'h:abc' and '[::1' is now one generic malformed-or-out-of-range message. 'user:pass@h' with _port is still refused.

Co-authored-by: Isaac <no-reply@databricks.com>
Signed-off-by: Cathleen Yan <58714163+cathleeny@users.noreply.github.com>
_is_bare_ipv6 now returns early unless the text has at least two colons and no leading bracket. ip_address raised a ValueError for every host name and host:port string, which made the folding path about 2x slower than bc7c09e. A bare IPv6 literal always has two or more colons, so behavior is unchanged; the session tests and the realkernel loopback tests pass.

Microbenchmark, median us per call (20000 calls x 7 repeats):
- bare host, _port 8443: 1.305 base, 1.261 new
- host:port, _port 8443: 1.364 base, 1.393 new (+2%)
- IPv6 bracketed, _port 8443: 1.340 base, 1.355 new (+1%)
- IPv6 bare, _port 8443: 1.413 base, 2.642 new (+87%, the successful ip_address parse)

Co-authored-by: Isaac <no-reply@databricks.com>
Signed-off-by: Cathleen Yan <58714163+cathleeny@users.noreply.github.com>

@peco-review-bot peco-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔀 Cross-reviewed: two independent reviews (Claude Opus 5.5 · GPT-6.1 Sol), checked by a judge model (Claude Sonnet 5.5).

✅ No issues identified.

Remove _is_bare_ipv6 and the ipaddress import. Bare IPv6 hosts such as ::1 never worked on the kernel path: the kernel prepends https:// and its URL parser rejects an unbracketed IPv6 literal. IPv6 literals must be bracketed ([::1]).

With _port set, urlsplit reads everything after the first colon of a bare IPv6 literal as the port. That text always contains another colon, so the integer parse fails and the existing "malformed IPv6 literal or port" ValueError is raised before any request. This includes _port=443. Without _port the host still passes through unchanged, as on main.

Tests: the ("::1", 8443) -> "[::1]:8443" row is removed. ::1 with _port 8443 or 443, and 2001:db8::1 with _port 8443, are now in the reject table.

Co-authored-by: Isaac <no-reply@databricks.com>
Signed-off-by: Cathleen Yan <58714163+cathleeny@users.noreply.github.com>

@peco-review-bot peco-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔀 Cross-reviewed: two independent reviews (Claude Opus 5.5 · GPT-6.1 Sol), checked by a judge model (Claude Sonnet 5.5).

✅ No issues identified.

🔀 1 finding considered and dismissed by the judge
  • 🟡 Medium src/databricks/sql/session.py:240 — An access_token combined with identity_federation_client_id bypasses the origin guard when its issuer matches server_hostname, although the bridge forwards that
    Judge's evidence: if routed_to_pat or not oauth:

This branch was previously deployed

1 inactive deployment
azure-prod — 5082c7ee Deployed Nov 20, 2026 by cathleeny via test-with-coverage #1337
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants