Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
42 commits
Select commit Hold shift + click to select a range
0ff5094
Complete Volume file operations through the native kernel
aakash-saravanan-db Nov 13, 2026
856e533
Consume Volume instructions after failed completion
aakash-saravanan-db Nov 13, 2026
53d8288
Fetch completed async results without a redundant status poll
aakash-saravanan-db Nov 13, 2026
42d3106
Clarify asynchronous Volume completion
aakash-saravanan-db Nov 13, 2026
5d1e6ab
Pin the native source for Volume completion
aakash-saravanan-db Nov 14, 2026
e8c9803
Document completed async result retrieval
aakash-saravanan-db Nov 14, 2026
68380a3
Select the Windows-compatible Volume kernel
aakash-saravanan-db Nov 14, 2026
d55977d
Keep async result reuse in the kernel and tolerate missing cancellers
aakash-saravanan-db Nov 14, 2026
3253170
Pin native Volume storage retry and terminal response fixes
aakash-saravanan-db Nov 14, 2026
2c8d0f7
Preserve Volume error provenance and stream teardown
aakash-saravanan-db Nov 14, 2026
63b22d2
Avoid cancelling completed Volume transfers during close
aakash-saravanan-db Nov 14, 2026
2031485
Format Volume cancellation cleanup comprehension
aakash-saravanan-db Nov 14, 2026
5027e63
Select native Volume inactivity timeout correction
aakash-saravanan-db Nov 14, 2026
060b64b
build(kernel): select corrected Volume completion
aakash-saravanan-db Nov 14, 2026
a689d26
build: select native Volume cancellation and error handling
aakash-saravanan-db Nov 15, 2026
5d728ca
build: use native root authorization fix
aakash-saravanan-db Nov 15, 2026
b93ade0
build: update kernel source for replayable uploads
aakash-saravanan-db Nov 15, 2026
3c83e9c
build: update kernel for local stream close
aakash-saravanan-db Nov 15, 2026
b6b20dc
fix(volume): select replayable file uploads and tighten validation
aakash-saravanan-db Nov 15, 2026
94f0a87
fix(kernel): use permission-preserving Volume downloads
aakash-saravanan-db Nov 16, 2026
8a0a433
build(kernel): select sanitized storage redirects
aakash-saravanan-db Nov 16, 2026
5965f81
Select native Volume lifetime and replay implementation
aakash-saravanan-db Nov 16, 2026
f14d16e
Pin kernel with statement lifecycle cleanup
aakash-saravanan-db Nov 16, 2026
094c214
Document Volume handles and verify async owner cleanup
aakash-saravanan-db Nov 16, 2026
39a6dc7
build: select kernel with async result cleanup
aakash-saravanan-db Nov 16, 2026
18c1a5a
build: select kernel with Volume instruction error telemetry
aakash-saravanan-db Nov 16, 2026
9c8d3e9
build: pin kernel with complete Volume instruction handling
aakash-saravanan-db Nov 16, 2026
d7d606d
build: use corrected kernel volume completion
aakash-saravanan-db Nov 17, 2026
827971a
build: select kernel with origin-safe Volume redirects
aakash-saravanan-db Nov 17, 2026
bef1659
build: select kernel with normalized upload lengths
aakash-saravanan-db Nov 17, 2026
3694957
build: pin kernel completion telemetry correction
aakash-saravanan-db Nov 17, 2026
974bf84
build: select kernel with restricted-directory GET support
aakash-saravanan-db Nov 18, 2026
cb95a26
Keep distinct Volume ownership test cases
aakash-saravanan-db Nov 18, 2026
bc2f439
build: pin kernel with bounded symlink resolution
aakash-saravanan-db Nov 18, 2026
55e7f74
Document kernel connection-close cancellation
aakash-saravanan-db Nov 18, 2026
f90c814
refactor(volume): streamline completion ownership
aakash-saravanan-db Nov 19, 2026
ab50ea0
build: select updated Volume kernel
aakash-saravanan-db Nov 19, 2026
b5a9035
fix(volume): preserve Python errors and transfer telemetry
aakash-saravanan-db Nov 20, 2026
b9a751b
fixup! fix(volume): preserve Python errors and transfer telemetry
aakash-saravanan-db Nov 20, 2026
0586423
fixup! fix(volume): preserve Python errors and transfer telemetry
aakash-saravanan-db Nov 20, 2026
8caaaf5
fixup! fix(volume): preserve Python errors and transfer telemetry
aakash-saravanan-db Nov 20, 2026
b05d562
fixup! fix(volume): preserve Python errors and transfer telemetry
aakash-saravanan-db Nov 20, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@

# Unreleased
- Honor `max_download_threads` on kernel connections, with a default download window of 16 and explicit chunk-setting overrides. Invalid values raise `ValueError` when connecting with `use_kernel=True`.
- Support Volume `PUT`, `GET` and `REMOVE` on the kernel backend, including existing Python stream uploads. Local file paths and resolved symlink targets must stay within configured allowed roots.
- Fix the kernel backend ignoring `use_cloud_fetch=False`, so complete results can be fetched without cloud-storage downloads.
- Fix inline parameters (`use_inline_params=True`) so a sequence directly after `IN` renders as a list again: `x IN %(ids)s` renders as `x IN (1,2,3)` instead of `x IN ARRAY(1,2,3)`, which Spark rejects. Inside the list, tuples render as rows for a multi-column `IN`, and a `set` or `frozenset` is accepted again. Sequences anywhere else still render as `ARRAY(...)` ([#629](https://github.com/databricks/databricks-sql-python/issues/629))
- Speed up Thrift response handling: read replies in 1 MiB chunks and decode them with Thrift's C-accelerated protocol, and stop stringifying every request/response for the default no-op debug logger. Wide result schemas, metadata calls with many rows, and inline Arrow results benefit most.
Expand Down
22 changes: 15 additions & 7 deletions CONNECTION_PARAMETERS.md
Original file line number Diff line number Diff line change
Expand Up @@ -114,7 +114,7 @@ to change without notice.

| Option | Type | Thrift | Kernel | Default Value | Note |
| ------------------------------------ | ----------- | :----: | :----: | ------------- | ---------------------------------------------------------------------------------------------------------------------------------------------- |
| `_socket_timeout` | `float` (s) | ✅ | ✅ | `900` (Thrift); `120` (kernel) | Thrift: socket send/recv/connect timeout. Kernel: total HTTP request deadline from connect through response-body completion. A positive value is forwarded; unset or `0` selects the kernel's 120s default. On the kernel path, `0` is neither unlimited nor an immediate timeout. |
| `_socket_timeout` | `float` (s) | ✅ | ✅ | `900` (Thrift SQL); `120` (kernel SQL) | Thrift: socket send/recv/connect timeout. Kernel: total HTTP request deadline from connect through response-body completion. A positive value is forwarded; unset or `0` selects the kernel's 120s SQL request default. Volume transfers have no total deadline: an explicit positive value caps storage connection establishment, reads, upload progress and PUT response waits. A redirected stream PUT resend has no idle bound. Unset or `0` leaves storage inactivity unbounded. |
| `_pool_connections` | `int` | ⚠️ | ❌ | `10` | Number of pools in the shared Python HTTP client. On Thrift this affects auxiliary auth/feature-flag/telemetry traffic, not the separate Thrift query transport. Kernel-owned network traffic does not use this client. |
| `_pool_maxsize` | `int` | ⚠️ | ⚠️ | `20` (Python); `100` (kernel when unset) | Max idle connections retained per host. It configures the shared Python client but not the Thrift query transport. With kernel ≥ 1.1.0, a positive value also configures the kernel's Rust HTTP pool. Unset or `0` keeps each client's default. |
| `_proxy_auth_method` | `str` | ✅ | ❌ | `None` | `basic` or `negotiate` (Kerberos). Applied to both Thrift query traffic and the shared Python client. It is not forwarded to the kernel's Rust transport. See [`docs/proxy.md`](docs/proxy.md). |
Expand Down Expand Up @@ -171,7 +171,7 @@ to change without notice.
| `use_inline_params` | `bool` \| `"silent"` | ✅ | ✅ | `False` | Render parameters inline (legacy) vs. native bound params (DBR 14.1+). Shared cursor code performs the rendering before dispatch, so both backends honor it identically. |
| `ignore_transactions` | `bool` | ✅ | ✅ | `True` | When `True`: `commit()` is a no-op, `rollback()` raises `NotSupportedError`, and setting `autocommit` is a no-op. |
| `fetch_autocommit_from_server`| `bool` | ✅ | ✅ | `False` | Query the server (`SET AUTOCOMMIT`) for autocommit state instead of returning the cached value. |
| `staging_allowed_local_path` | `str` \| `List[str]` | ✅ | ❌ | `None` | Local path(s) permitted for Unity Catalog Volume `PUT`/`GET`. **Thrift-only** — the kernel has no Volume API yet. |
| `staging_allowed_local_path` | `str` \| `List[str]` | ✅ | ✅ | `None` | Local path(s) permitted for Unity Catalog Volume `PUT`/`GET`. Kernel also checks resolved symlink targets; see below. |

## Telemetry

Expand Down Expand Up @@ -208,13 +208,12 @@ are forwarded when supported by the installed kernel binding.
`enable_query_result_lz4_compression`.
5. Arrow-native rendering for `_use_arrow_native_decimals` /
`_use_arrow_native_timestamps` (complex types **are** forwarded).
6. `staging_allowed_local_path` (Volume `PUT`/`GET`).
7. Retry fine-tuning beyond the four forwarded knobs: `_retry_delay_default`,
6. Retry fine-tuning beyond the four forwarded knobs: `_retry_delay_default`,
`_retry_dangerous_codes`, `_respect_server_retry_after_header`,
`_retry_max_redirects`, `_enable_v3_retries`.
8. `force_enable_telemetry`.
9. `_port` and `_connection_uri`.
10. `_pool_connections` and `_proxy_auth_method` do not configure the
7. `force_enable_telemetry`.
8. `_port` and `_connection_uri`.
9. `_pool_connections` and `_proxy_auth_method` do not configure the
kernel-owned network stack.

### Supported on Kernel, missing / ignored on Thrift
Expand Down Expand Up @@ -246,3 +245,12 @@ are forwarded when supported by the installed kernel binding.
> subject to change. Generated against connector version `4.5.0`; the source of
> truth is `Connection.__init__` (`src/databricks/sql/client.py`) and the
> per-backend clients under `src/databricks/sql/backend/`.

### Volume file paths

With `use_kernel=True`, SQL `PUT`, `GET` and `REMOVE` complete their file
operations before `execute()` or `get_async_execution_result()` returns.
`PUT` and `GET` require requested paths and resolved symlink targets within
`staging_allowed_local_path`; `REMOVE` needs no local root.
`execute(input_stream=stream)` reads the stream only when requested by the server.
Completed transfers return no instruction rows.
2 changes: 1 addition & 1 deletion KERNEL_REV
Original file line number Diff line number Diff line change
@@ -1 +1 @@
0b5d0edfed32c0745d5abc2468fc3070e2eb2a17
c2ddddc61bd41d9529dd19f36546d603ba21150d
8 changes: 7 additions & 1 deletion src/databricks/sql/backend/kernel/_errors.py
Original file line number Diff line number Diff line change
Expand Up @@ -141,7 +141,13 @@ def reraise_kernel_error(exc: "_kernel.KernelError") -> "Error":
"operation-id": getattr(exc, "query_id", None),
"diagnostic-info": getattr(exc, "diagnostic_info", None),
}
new = cls(getattr(exc, "message", str(exc)), context)
message = getattr(exc, "message", str(exc))
if getattr(exc, "error_code", None) == "VOLUME_STORAGE_HTTP_ERROR":
cls = OperationalError
message = "Staging operation over HTTP was unsuccessful: " + message.replace(
"Volume storage request failed: HTTP ", "", 1
)
new = cls(message, context)

for attr in (
"code",
Expand Down
98 changes: 21 additions & 77 deletions src/databricks/sql/backend/kernel/client.py
Original file line number Diff line number Diff line change
Expand Up @@ -16,8 +16,6 @@

- ``query_tags`` on execute is not supported (kernel exposes
``statement_conf`` but PyO3 doesn't surface it).
- Volume PUT/GET (staging operations): kernel has no Volume API
yet. Users on Thrift-only paths.
"""

from __future__ import annotations
Expand Down Expand Up @@ -84,41 +82,6 @@
# per-request skip-and-warn.
_KERNEL_MANAGED_HEADERS = frozenset({"authorization", "x-databricks-org-id"})

# Leading verbs of SQL volume/staging statements. Detected by the
# leading token (case-insensitive) so the kernel backend can fail loud
# on staging ops it can't service — see ``execute_command``.
_STAGING_VERBS = ("PUT", "GET", "REMOVE")


def _strip_leading_sql_comments(sql: str) -> str:
"""Strip leading whitespace and SQL comments (``-- …`` line and
``/* … */`` block, possibly several) from ``sql``, returning the
remainder.

Needed so staging detection sees the real leading verb: a
comment-prefixed staging op (``-- upload\\nPUT …`` or
``/* c */ PUT …``, common in ETL scripts) must still be classified
as staging, or it would slip past the guard into the silent-no-op
bug. Block comments do not nest in Databricks SQL, so a simple
scan-to-``*/`` is correct.
"""
i = 0
n = len(sql)
while i < n:
if sql[i].isspace():
i += 1
elif sql.startswith("--", i):
# Line comment: skip to end of line (or string).
nl = sql.find("\n", i)
i = n if nl == -1 else nl + 1
elif sql.startswith("/*", i):
# Block comment: skip to closing */ (or end if unterminated).
close = sql.find("*/", i + 2)
i = n if close == -1 else close + 2
else:
break
return sql[i:]


def _is_not_found(exc: BaseException) -> bool:
"""True iff ``exc`` is a kernel ``NotFound`` error (HTTP 404 /
Expand Down Expand Up @@ -155,21 +118,6 @@ def _catalog_or_none(value: Optional[str]) -> Optional[str]:
return value


def _is_staging_statement(operation: str) -> bool:
"""True iff ``operation`` is a volume/staging statement (PUT / GET /
REMOVE).

Strips leading whitespace + SQL comments first (so a comment-
prefixed staging op is still caught), then matches the leading token
only — so a normal query that merely *contains* the word (e.g.
``SELECT 'GET' AS x``) isn't misflagged.
"""
stripped = _strip_leading_sql_comments(operation)
# First whitespace-delimited token, uppercased.
verb = stripped.split(None, 1)[0].upper() if stripped.strip() else ""
return verb in _STAGING_VERBS


class _KernelSessionParameterSupport(NamedTuple):
session_type: Any
parameters: FrozenSet[str]
Expand Down Expand Up @@ -443,6 +391,8 @@ def __init__(
# Guarded by its own lock — cancel can race execute teardown.
self._sync_cancellers: Dict[int, Any] = {}
self._sync_cancellers_lock = threading.RLock()
# Volume transfers remain cancellable after synchronous SQL execution returns.
self._volume_handles: Dict[str, Any] = {}

# ── Session lifecycle ──────────────────────────────────────────

Expand Down Expand Up @@ -643,24 +593,6 @@ def execute_command(
# ``_async_statements`` and closed by ``close_command``); the sync
# path drops it in finally. ``close_stmt`` is the post-success
# decision flag — it stays True on sync, flips to False on async.
# Volume/staging (PUT/GET/REMOVE) is not supported on the kernel
# path: the kernel returns the staging control row as a normal
# result set (``KernelResultSet.is_staging_operation`` is always
# False), so the connector's ``_handle_staging_operation`` never
# fires and NO file is transferred. Rather than silently no-op
# (the Thrift path performs the presigned-URL upload/download),
# fail loud at the call site so ETL scripts don't ingest
# stale/missing data. Detected by the leading SQL verb — the
# only signal available pre-execute, since the kernel exposes no
# staging marker today.
if _is_staging_statement(operation):
raise NotSupportedError(
"Volume / staging operations (PUT / GET / REMOVE) are not "
"supported on the kernel backend (use_kernel=True); the file "
"transfer would silently not happen. Use the Thrift backend "
"for staging operations."
)

close_stmt = True
try:
try:
Expand Down Expand Up @@ -702,6 +634,12 @@ def execute_command(
# Canceller is best-effort; never block execute on it.
pass
executed = stmt.execute()
command_id = CommandId.from_sea_statement_id(executed.statement_id)
if executed.is_volume_operation:

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟠 High — pyproject.toml:73 still allows databricks-sql-kernel ^1.4.0, but the code now calls three kernel APIs that don't exist yet: ExecutedStatement.is_volume_operation (read here on every sync execute), kernel_handle.is_volume_operation (read in KernelResultSet.__init__ for every result, including metadata ResultStreams), and fetch_result() in place of await_result() for async results.

If the resolver installs any wheel in the allowed range that lacks them, this isn't limited to Volume ops. Every execute(), every metadata call and every get_async_execution_result() raises AttributeError, so ordinary SELECT 1 breaks on use_kernel=True. The PR description says the floor "has not been selected". This should block merge: bump the floor in the same change. Until then, reading is_volume_operation with getattr(..., False) would at least keep non-volume queries working.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The optional-canceller defect is corrected at ab1efce1: registration uses .get, the Statement remains retained, and cleanup skips absent cancellers. All 276 affected tests pass; final native live checks preserve lazy BinaryIO and one-read empty/nonempty streams.

The new Volume marker/completion API dependency is a valid merge prerequisite. The source build pins compatible 22f96fac; a compatible published native version/floor and installed-package proof remain explicitly open. This PR is not merge-ready, and no old-native fallback or invented release version was added.

The fetch_result subclaim is separate: released 1.4.0 exposes it, but the substitution is removed from this PR. Merged native #451 provides terminal-response reuse with the original await_result boundary. Actual public-cursor owner and reader/refetch tests show identical await/fetch requests, rows and metadata on the selected native; redundant #1056 is superseded without merge.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for confirming the Volume-marker dependency is a merge prerequisite, and for dropping the fetch_result() substitution. That part of the finding is resolved. The core issue is still there, though. pyproject.toml:73 still says databricks-sql-kernel ^1.4.0. Line 641 still reads executed.is_volume_operation with no guard. KernelResultSet.__init__ still reads kernel_handle.is_volume_operation the same way. So if a published wheel in that range lacks the marker, ordinary sync executes and metadata calls fail with AttributeError before the code ever checks for a Volume op. A source build pinned to 22f96fac doesn't limit what the resolver installs. Leaving this open until the floor is raised to the first published wheel that ships the marker, or until both reads are guarded so non-volume queries keep working.

Comment thread
aakash-saravanan-db marked this conversation as resolved.
Resolved
Comment thread
aakash-saravanan-db marked this conversation as resolved.
Resolved
Comment thread
aakash-saravanan-db marked this conversation as resolved.
Resolved
Comment thread
aakash-saravanan-db marked this conversation as resolved.
Resolved
Comment thread
aakash-saravanan-db marked this conversation as resolved.
Resolved
Comment thread
aakash-saravanan-db marked this conversation as resolved.
Resolved

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟠 High — The PR reads executed.is_volume_operation and kernel_handle.is_volume_operation unconditionally, but pyproject.toml and poetry.lock still allow kernel 1.4.0. That release almost certainly lacks the Volume API, since the PR only bumps KERNEL_REV (CI source). Installed wheels could then fail every query with AttributeError. Raise the floor and lock to the release that ships the Volume API, or read the attribute with getattr(..., False).

🔀 Found by both reviewers · confirmed by the judge

with self._sync_cancellers_lock:
self._volume_handles[command_id.guid] = (
self._sync_cancellers.get(id(cursor))
)
# Execute succeeded: the kernel now owns the statement
# lifecycle. It auto-closes the server statement when the
# result stream is fully drained (``ExecutedStatement::
Expand Down Expand Up @@ -751,7 +689,6 @@ def execute_command(
except Exception:
pass

command_id = CommandId.from_sea_statement_id(executed.statement_id)
# Surface the affected-row count for DML (INSERT/UPDATE/DELETE/
# MERGE) as ``cursor.rowcount`` instead of the hardcoded ``-1``.
# ``num_modified_rows`` is ``None`` for SELECT (and warehouses
Expand All @@ -774,16 +711,21 @@ def execute_command(
try:
return self._make_result_set(executed, cursor, command_id)
except Exception as exc:
self.finish_volume_operation(command_id)
raise _wrap_kernel_exception("execute_command", exc) from exc

def finish_volume_operation(self, command_id: CommandId) -> None:
with self._sync_cancellers_lock:
self._volume_handles.pop(command_id.guid, None)

def cancel_command(self, command_id: CommandId) -> None:
with self._async_handles_lock:
handle = self._async_handles.get(command_id.guid)
with self._sync_cancellers_lock:
handle = self._volume_handles.get(command_id.guid)
if handle is None:
# Sync-execute paths fully materialise the result before
# ``execute_command`` returns, so by the time
# cancel_command can fire there's nothing in flight.
# Match the Thrift backend's tolerant behaviour.
with self._async_handles_lock:
handle = self._async_handles.get(command_id.guid)
if handle is None:
# Completed synchronous operations have no retained canceller.
logger.debug("cancel_command: no in-flight async handle for %s", command_id)
return
try:
Expand Down Expand Up @@ -832,6 +774,8 @@ def cancel_running_cursor(self, cursor: "Cursor") -> bool:
return True

def close_command(self, command_id: CommandId) -> None:
with self._sync_cancellers_lock:
self._volume_handles.pop(command_id.guid, None)
with self._async_handles_lock:
handle = self._async_handles.pop(command_id.guid, None)
stmt = self._async_statements.pop(command_id.guid, None)
Expand Down
Loading
Loading