Repository navigation
Complete Volume file operations through the kernel backend #1057
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Changes from all commits
0ff5094
856e533
53d8288
42d3106
5d1e6ab
e8c9803
68380a3
d55977d
3253170
2c8d0f7
63b22d2
2031485
5027e63
060b64b
a689d26
5d728ca
b93ade0
3c83e9c
b6b20dc
94f0a87
8a0a433
5965f81
f14d16e
094c214
39a6dc7
18c1a5a
9c8d3e9
d7d606d
827971a
bef1659
3694957
974bf84
cb95a26
bc2f439
55e7f74
f90c814
ab50ea0
b5a9035
b9a751b
0586423
8caaaf5
b05d562
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1 +1 @@ | ||
| 0b5d0edfed32c0745d5abc2468fc3070e2eb2a17 | ||
| c2ddddc61bd41d9529dd19f36546d603ba21150d |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -16,8 +16,6 @@ | |
|
|
||
| - ``query_tags`` on execute is not supported (kernel exposes | ||
| ``statement_conf`` but PyO3 doesn't surface it). | ||
| - Volume PUT/GET (staging operations): kernel has no Volume API | ||
| yet. Users on Thrift-only paths. | ||
| """ | ||
|
|
||
| from __future__ import annotations | ||
|
|
@@ -84,41 +82,6 @@ | |
| # per-request skip-and-warn. | ||
| _KERNEL_MANAGED_HEADERS = frozenset({"authorization", "x-databricks-org-id"}) | ||
|
|
||
| # Leading verbs of SQL volume/staging statements. Detected by the | ||
| # leading token (case-insensitive) so the kernel backend can fail loud | ||
| # on staging ops it can't service — see ``execute_command``. | ||
| _STAGING_VERBS = ("PUT", "GET", "REMOVE") | ||
|
|
||
|
|
||
| def _strip_leading_sql_comments(sql: str) -> str: | ||
| """Strip leading whitespace and SQL comments (``-- …`` line and | ||
| ``/* … */`` block, possibly several) from ``sql``, returning the | ||
| remainder. | ||
|
|
||
| Needed so staging detection sees the real leading verb: a | ||
| comment-prefixed staging op (``-- upload\\nPUT …`` or | ||
| ``/* c */ PUT …``, common in ETL scripts) must still be classified | ||
| as staging, or it would slip past the guard into the silent-no-op | ||
| bug. Block comments do not nest in Databricks SQL, so a simple | ||
| scan-to-``*/`` is correct. | ||
| """ | ||
| i = 0 | ||
| n = len(sql) | ||
| while i < n: | ||
| if sql[i].isspace(): | ||
| i += 1 | ||
| elif sql.startswith("--", i): | ||
| # Line comment: skip to end of line (or string). | ||
| nl = sql.find("\n", i) | ||
| i = n if nl == -1 else nl + 1 | ||
| elif sql.startswith("/*", i): | ||
| # Block comment: skip to closing */ (or end if unterminated). | ||
| close = sql.find("*/", i + 2) | ||
| i = n if close == -1 else close + 2 | ||
| else: | ||
| break | ||
| return sql[i:] | ||
|
|
||
|
|
||
| def _is_not_found(exc: BaseException) -> bool: | ||
| """True iff ``exc`` is a kernel ``NotFound`` error (HTTP 404 / | ||
|
|
@@ -155,21 +118,6 @@ def _catalog_or_none(value: Optional[str]) -> Optional[str]: | |
| return value | ||
|
|
||
|
|
||
| def _is_staging_statement(operation: str) -> bool: | ||
| """True iff ``operation`` is a volume/staging statement (PUT / GET / | ||
| REMOVE). | ||
|
|
||
| Strips leading whitespace + SQL comments first (so a comment- | ||
| prefixed staging op is still caught), then matches the leading token | ||
| only — so a normal query that merely *contains* the word (e.g. | ||
| ``SELECT 'GET' AS x``) isn't misflagged. | ||
| """ | ||
| stripped = _strip_leading_sql_comments(operation) | ||
| # First whitespace-delimited token, uppercased. | ||
| verb = stripped.split(None, 1)[0].upper() if stripped.strip() else "" | ||
| return verb in _STAGING_VERBS | ||
|
|
||
|
|
||
| class _KernelSessionParameterSupport(NamedTuple): | ||
| session_type: Any | ||
| parameters: FrozenSet[str] | ||
|
|
@@ -443,6 +391,8 @@ def __init__( | |
| # Guarded by its own lock — cancel can race execute teardown. | ||
| self._sync_cancellers: Dict[int, Any] = {} | ||
| self._sync_cancellers_lock = threading.RLock() | ||
| # Volume transfers remain cancellable after synchronous SQL execution returns. | ||
| self._volume_handles: Dict[str, Any] = {} | ||
|
|
||
| # ── Session lifecycle ────────────────────────────────────────── | ||
|
|
||
|
|
@@ -643,24 +593,6 @@ def execute_command( | |
| # ``_async_statements`` and closed by ``close_command``); the sync | ||
| # path drops it in finally. ``close_stmt`` is the post-success | ||
| # decision flag — it stays True on sync, flips to False on async. | ||
| # Volume/staging (PUT/GET/REMOVE) is not supported on the kernel | ||
| # path: the kernel returns the staging control row as a normal | ||
| # result set (``KernelResultSet.is_staging_operation`` is always | ||
| # False), so the connector's ``_handle_staging_operation`` never | ||
| # fires and NO file is transferred. Rather than silently no-op | ||
| # (the Thrift path performs the presigned-URL upload/download), | ||
| # fail loud at the call site so ETL scripts don't ingest | ||
| # stale/missing data. Detected by the leading SQL verb — the | ||
| # only signal available pre-execute, since the kernel exposes no | ||
| # staging marker today. | ||
| if _is_staging_statement(operation): | ||
| raise NotSupportedError( | ||
| "Volume / staging operations (PUT / GET / REMOVE) are not " | ||
| "supported on the kernel backend (use_kernel=True); the file " | ||
| "transfer would silently not happen. Use the Thrift backend " | ||
| "for staging operations." | ||
| ) | ||
|
|
||
| close_stmt = True | ||
| try: | ||
| try: | ||
|
|
@@ -702,6 +634,12 @@ def execute_command( | |
| # Canceller is best-effort; never block execute on it. | ||
| pass | ||
| executed = stmt.execute() | ||
| command_id = CommandId.from_sea_statement_id(executed.statement_id) | ||
| if executed.is_volume_operation: | ||
|
aakash-saravanan-db marked this conversation as resolved.
Resolved
aakash-saravanan-db marked this conversation as resolved.
Resolved
aakash-saravanan-db marked this conversation as resolved.
Resolved
aakash-saravanan-db marked this conversation as resolved.
Resolved
aakash-saravanan-db marked this conversation as resolved.
Resolved
aakash-saravanan-db marked this conversation as resolved.
Resolved
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🟠 High — The PR reads 🔀 Found by both reviewers · confirmed by the judge |
||
| with self._sync_cancellers_lock: | ||
| self._volume_handles[command_id.guid] = ( | ||
| self._sync_cancellers.get(id(cursor)) | ||
| ) | ||
| # Execute succeeded: the kernel now owns the statement | ||
| # lifecycle. It auto-closes the server statement when the | ||
| # result stream is fully drained (``ExecutedStatement:: | ||
|
|
@@ -751,7 +689,6 @@ def execute_command( | |
| except Exception: | ||
| pass | ||
|
|
||
| command_id = CommandId.from_sea_statement_id(executed.statement_id) | ||
| # Surface the affected-row count for DML (INSERT/UPDATE/DELETE/ | ||
| # MERGE) as ``cursor.rowcount`` instead of the hardcoded ``-1``. | ||
| # ``num_modified_rows`` is ``None`` for SELECT (and warehouses | ||
|
|
@@ -774,16 +711,21 @@ def execute_command( | |
| try: | ||
| return self._make_result_set(executed, cursor, command_id) | ||
| except Exception as exc: | ||
| self.finish_volume_operation(command_id) | ||
| raise _wrap_kernel_exception("execute_command", exc) from exc | ||
|
|
||
| def finish_volume_operation(self, command_id: CommandId) -> None: | ||
| with self._sync_cancellers_lock: | ||
| self._volume_handles.pop(command_id.guid, None) | ||
|
|
||
| def cancel_command(self, command_id: CommandId) -> None: | ||
| with self._async_handles_lock: | ||
| handle = self._async_handles.get(command_id.guid) | ||
| with self._sync_cancellers_lock: | ||
| handle = self._volume_handles.get(command_id.guid) | ||
| if handle is None: | ||
| # Sync-execute paths fully materialise the result before | ||
| # ``execute_command`` returns, so by the time | ||
| # cancel_command can fire there's nothing in flight. | ||
| # Match the Thrift backend's tolerant behaviour. | ||
| with self._async_handles_lock: | ||
| handle = self._async_handles.get(command_id.guid) | ||
| if handle is None: | ||
| # Completed synchronous operations have no retained canceller. | ||
| logger.debug("cancel_command: no in-flight async handle for %s", command_id) | ||
| return | ||
| try: | ||
|
|
@@ -832,6 +774,8 @@ def cancel_running_cursor(self, cursor: "Cursor") -> bool: | |
| return True | ||
|
|
||
| def close_command(self, command_id: CommandId) -> None: | ||
| with self._sync_cancellers_lock: | ||
| self._volume_handles.pop(command_id.guid, None) | ||
| with self._async_handles_lock: | ||
| handle = self._async_handles.pop(command_id.guid, None) | ||
| stmt = self._async_statements.pop(command_id.guid, None) | ||
|
|
||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
🟠 High —
pyproject.toml:73still allowsdatabricks-sql-kernel ^1.4.0, but the code now calls three kernel APIs that don't exist yet:ExecutedStatement.is_volume_operation(read here on every sync execute),kernel_handle.is_volume_operation(read inKernelResultSet.__init__for every result, including metadataResultStreams), andfetch_result()in place ofawait_result()for async results.If the resolver installs any wheel in the allowed range that lacks them, this isn't limited to Volume ops. Every
execute(), every metadata call and everyget_async_execution_result()raisesAttributeError, so ordinarySELECT 1breaks onuse_kernel=True. The PR description says the floor "has not been selected". This should block merge: bump the floor in the same change. Until then, readingis_volume_operationwithgetattr(..., False)would at least keep non-volume queries working.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The optional-canceller defect is corrected at
ab1efce1: registration uses.get, the Statement remains retained, and cleanup skips absent cancellers. All 276 affected tests pass; final native live checks preserve lazy BinaryIO and one-read empty/nonempty streams.The new Volume marker/completion API dependency is a valid merge prerequisite. The source build pins compatible
22f96fac; a compatible published native version/floor and installed-package proof remain explicitly open. This PR is not merge-ready, and no old-native fallback or invented release version was added.The fetch_result subclaim is separate: released 1.4.0 exposes it, but the substitution is removed from this PR. Merged native #451 provides terminal-response reuse with the original await_result boundary. Actual public-cursor owner and reader/refetch tests show identical await/fetch requests, rows and metadata on the selected native; redundant #1056 is superseded without merge.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for confirming the Volume-marker dependency is a merge prerequisite, and for dropping the
fetch_result()substitution. That part of the finding is resolved. The core issue is still there, though.pyproject.toml:73still saysdatabricks-sql-kernel ^1.4.0. Line 641 still readsexecuted.is_volume_operationwith no guard.KernelResultSet.__init__still readskernel_handle.is_volume_operationthe same way. So if a published wheel in that range lacks the marker, ordinary sync executes and metadata calls fail withAttributeErrorbefore the code ever checks for a Volume op. A source build pinned to22f96facdoesn't limit what the resolver installs. Leaving this open until the floor is raised to the first published wheel that ships the marker, or until both reads are guarded so non-volume queries keep working.