Skip to content

Complete Volume file operations through the kernel backend - #1057

Open
aakash-saravanan-db wants to merge 42 commits into
mainfrom
aakash-saravanan_data/feat/volume-restart-20261113
Open

aakash-saravanan-db wants to merge 42 commits into
mainfrom
aakash-saravanan_data/feat/volume-restart-20261113

Conversation

@aakash-saravanan-db

@aakash-saravanan-db aakash-saravanan-db commented Nov 13, 2026 •

Copy link
Copy Markdown
Collaborator

Feature or gap

Kernel-backed cursors need to complete server-marked Volume transfers through the Python staging API.

What this PR addresses

Complete file PUT, GET, REMOVE and BinaryIO PUT through the native core. Keep the mutable statement owner and detached canceller available during synchronous transfer, consume instruction rows on completion, and preserve Python filesystem and HTTP exception mapping. Async execution completes through its retained result stream.

Dependencies

Requires the kernel Volume APIs in databricks/databricks-sql-kernel#465 and a compatible coordinated native release. The package floor and lock must select that release, with clean installed-driver validation before merge.

Kernel version

The compatible kernel release has not been selected. When it is available, update the dependency floor and lockfile in this PR and validate a clean installation before merge. KERNEL_REV selects source for local and CI builds only.

Tests and validation

The focused client and result-set suites pass: 292 tests. Local validation uses source-built native inputs; package installation and scheduled activation remain separate gates.

@peco-review-bot peco-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: 1 High · 1 Medium · 1 Low

Needs changes. The main blocker is that the code relies on new kernel APIs while the dependency floor still allows older wheels, which would break every kernel query, not only Volume operations. There is also a KeyError crash when the best-effort canceller is missing.

Other findings

  • 🔵 Low — This PR deletes the kernel e2e tests for staging and adds no replacements. All new coverage uses mocks that invent is_volume_operation, complete_volume_operation return strings ("needs_input_stream", "completed", "not_volume") and exception attributes (local_os_error, http_status, local_path). Nothing in CI checks that the real kernel binding actually has these names and values. Please add kernel e2e cases, such as PUT→GET→REMOVE round trips (file and input_stream) and a rejected symlink outside the allowed root, so a mismatch with the kernel contract fails the build.

# Canceller is best-effort; never block execute on it.
pass
executed = stmt.execute()
if executed.is_volume_operation:

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟠 High — pyproject.toml:73 still allows databricks-sql-kernel ^1.4.0, but the code now calls three kernel APIs that don't exist yet: ExecutedStatement.is_volume_operation (read here on every sync execute), kernel_handle.is_volume_operation (read in KernelResultSet.__init__ for every result, including metadata ResultStreams), and fetch_result() in place of await_result() for async results.

If the resolver installs any wheel in the allowed range that lacks them, this isn't limited to Volume ops. Every execute(), every metadata call and every get_async_execution_result() raises AttributeError, so ordinary SELECT 1 breaks on use_kernel=True. The PR description says the floor "has not been selected". This should block merge: bump the floor in the same change. Until then, reading is_volume_operation with getattr(..., False) would at least keep non-volume queries working.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The optional-canceller defect is corrected at ab1efce1: registration uses .get, the Statement remains retained, and cleanup skips absent cancellers. All 276 affected tests pass; final native live checks preserve lazy BinaryIO and one-read empty/nonempty streams.

The new Volume marker/completion API dependency is a valid merge prerequisite. The source build pins compatible 22f96fac; a compatible published native version/floor and installed-package proof remain explicitly open. This PR is not merge-ready, and no old-native fallback or invented release version was added.

The fetch_result subclaim is separate: released 1.4.0 exposes it, but the substitution is removed from this PR. Merged native #451 provides terminal-response reuse with the original await_result boundary. Actual public-cursor owner and reader/refetch tests show identical await/fetch requests, rows and metadata on the selected native; redundant #1056 is superseded without merge.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for confirming the Volume-marker dependency is a merge prerequisite, and for dropping the fetch_result() substitution. That part of the finding is resolved. The core issue is still there, though. pyproject.toml:73 still says databricks-sql-kernel ^1.4.0. Line 641 still reads executed.is_volume_operation with no guard. KernelResultSet.__init__ still reads kernel_handle.is_volume_operation the same way. So if a published wheel in that range lacks the marker, ordinary sync executes and metadata calls fail with AttributeError before the code ever checks for a Volume op. A source build pinned to 22f96fac doesn't limit what the resolver installs. Leaving this open until the floor is raised to the first published wheel that ships the marker, or until both reads are guarded so non-volume queries keep working.

@peco-review-bot peco-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: 1 High · 1 Medium · 2 Nit

Needs changes: 1 high, 1 medium and 2 low findings. The high one: the new is_volume_operation attribute is read unconditionally on every execute, but the kernel floor is still ^1.4.0, so released wheels would break all kernel queries. The medium one: any kernel error with an HTTP status is turned into a generic staging OperationalError. Follow-up already noted in the PR: Windows and installed released-package validation are still open.

Other findings

  • ⚪ Nit — The new ordering (backend close_command first, then _kernel_handle.close()) now applies to every kernel result set, not only Volume ones. For async results, close_command closes the retained owning handle and the parent Statement (client.py ~800-820) while the result stream taken from them is still open. Before this change the stream was closed first. Any error from closing the stream afterwards is logged and swallowed, but this changes the teardown order for all async results to work around a lock that only Volume transfers hold. Consider cancelling only the Volume canceller first and keeping the old order for everything else.
  • ⚪ Nit — Several new cancellation paths have no unit tests: close_session now cancels all in-flight sync cancellers and retained Volume cancellers, and close_command cancels a pending Volume canceller (client.py:784-790). The only Volume backend test covers cancel_command and finish_volume_operation. test_volume_close_signals_backend_before_native_lock uses a mocked backend, so it doesn't exercise the real close_command logic. Add tests that (a) close_command on a registered Volume id calls cancel() and removes the entry, and (b) close_session cancels and clears both maps, even when cancel() raises.

Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>
Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>
Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>
Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>
Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>
Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>
Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>
Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>
Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>
Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>
Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>
Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>
Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>
Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>
Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>
Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>
Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>
Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>
Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>
Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>
Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>
Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>
Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>

@peco-review-bot peco-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔀 Cross-reviewed: two independent reviews (Claude Opus 5.5 · GPT-6.1 Sol), checked by a judge model (Claude Sonnet 5.5).

Verdict: 1 High

Needs a dependency-floor fix: 1 medium issue. The completion, cancel and close lifecycle for sync and async Volume operations looks consistent. Nit: the new error-message strings in _complete_volume_operation go past the 100-character line limit.

results_queue=None,
description=description_from_arrow_schema(schema),
is_staging_operation=False,
is_staging_operation=kernel_handle.is_volume_operation,

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟠 High — KernelResultSet.__init__ reads kernel_handle.is_volume_operation unguarded, but pyproject still allows ^1.4.0. A kernel wheel without the Volume API would raise AttributeError on every query and metadata call. Raise the dependency floor to the release that adds the Volume API, or read the attribute with getattr(..., False).

🔀 Found by both reviewers · confirmed by the judge

Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>

@peco-review-bot peco-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔀 Cross-reviewed: two independent reviews (Claude Opus 5.5 · GPT-6.1 Sol), checked by a judge model (Claude Sonnet 5.5).

Verdict: 1 High

1 high concern. Completion, cancellation, and error mapping work as described and match Thrift's allowed-path and stream semantics. However, the new kernel attributes are read on every query, so this PR depends on the pending dependency-floor bump to avoid breaking all kernel queries.

results_queue=None,
description=description_from_arrow_schema(schema),
is_staging_operation=False,
is_staging_operation=kernel_handle.is_volume_operation,

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟠 High — KernelResultSet.__init__ reads kernel_handle.is_volume_operation for every result, but pyproject.toml still allows ^1.4.0 and the lock pins 1.4.0. The PR only bumps KERNEL_REV, which affects CI source, not the released wheel. An installed wheel without this API would raise AttributeError on every kernel query. Raise the floor and lock to the released wheel that ships is_volume_operation/VolumeCompletion, or read it with getattr(..., False).

🔀 Found by both reviewers · confirmed by the judge

Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>

@peco-review-bot peco-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔀 Cross-reviewed: two independent reviews (Claude Opus 5.5 · GPT-6.1 Sol), checked by a judge model (Claude Sonnet 5.5).

Verdict: 1 High

Needs changes: 1 high concern. The completion, cancellation and error-mapping flow looks correct and has tests. But because the packaging floor is not bumped, every kernel query breaks on wheels that are currently allowed.

pass
executed = stmt.execute()
command_id = CommandId.from_sea_statement_id(executed.statement_id)
if executed.is_volume_operation:

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟠 High — The PR reads executed.is_volume_operation and kernel_handle.is_volume_operation unconditionally, but pyproject.toml and poetry.lock still allow kernel 1.4.0. That release almost certainly lacks the Volume API, since the PR only bumps KERNEL_REV (CI source). Installed wheels could then fail every query with AttributeError. Raise the floor and lock to the release that ships the Volume API, or read the attribute with getattr(..., False).

🔀 Found by both reviewers · confirmed by the judge

Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>

@peco-review-bot peco-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔀 Cross-reviewed: two independent reviews (Claude Opus 5.5 · GPT-6.1 Sol), checked by a judge model (Claude Sonnet 5.5).

Verdict: 1 High

One high-severity blocker: the kernel dependency floor still allows wheels that lack the Volume API. The completion, cancellation and error-mapping logic matches the Thrift Volume path, and the unit tests cover the main outcomes.

results_queue=None,
description=description_from_arrow_schema(schema),
is_staging_operation=False,
is_staging_operation=kernel_handle.is_volume_operation,

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟠 High — The kernel result set and execute_command now read is_volume_operation unconditionally, but pyproject still allows ^1.4.0. A wheel without the Volume API would raise AttributeError on every kernel query. Raise the floor to the release that adds the Volume API and update the lock file.

🔀 Found by both reviewers · confirmed by the judge

This branch was previously deployed

1 inactive deployment
azure-prod — b05d5622 Deployed Nov 20, 2026 by peco-review-bot[bot] via followup #2582
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant