feat(kernel): surface kernel logs through Python logging on use_kernel (#824)

* feat(kernel): surface kernel logs through Python logging on use_kernel

When the kernel backend loads, auto-initialize the kernel's
tracing -> Python logging bridge so `use_kernel=True` users see kernel
logs with no extra setup. Kernel logs land under the
`databricks.sql.kernel` logger (a child of the connector's
`databricks.sql.*` namespace), so an existing
`logging.getLogger("databricks.sql").setLevel(...)` cascades to them.

- `_errors.py` calls `databricks_sql_kernel.init_logging()` once at
  extension load (it's the canonical kernel-import site). The call is
  `getattr`-guarded so an older kernel wheel without the function still
  works — just without kernel logs.
- e2e tests assert kernel records reach the `databricks.sql.kernel`
  logger (and the pyo3 boundary under `databricks.sql.kernel.pyo3`) and
  that the level set on the logger is respected. Creds-gated per the
  existing kernel e2e convention.

Requires the companion kernel/pyo3 change
(databricks-sql-kernel#120) that exposes `init_logging()`.

Co-authored-by: Isaac
Signed-off-by: Vikrant Puppala <vikrant.puppala@databricks.com>

* fix(kernel): harden log-bridge init + address review feedback

Review feedback from PR #824 (gopalldb):

- P1: guard the init_logging() call against throwing, not just a missing
  function. A panic across the PyO3 boundary surfaces as
  pyo3_runtime.PanicException (a BaseException, not Exception), so a bare
  call could escape module import and fail every use_kernel=True
  connection over a non-essential logging feature. Wrap in
  try/except BaseException, log a debug breadcrumb, continue. This also
  neutralizes the idempotency concern regardless of the Rust impl.
- P2: soften the level-control test docstring to make clear it asserts
  the effective customer-visible outcome (sub-threshold records don't
  surface), not source-side suppression — caplog filters after the FFI.
- P2: downgrade the databricks.sql.kernel.pyo3 assertion to a soft
  warning so a benign kernel change to the boundary breadcrumb target
  doesn't break the connector e2e suite. The core
  databricks.sql.kernel contract is still hard-asserted.

Co-authored-by: Isaac
Signed-off-by: Vikrant Puppala <vikrant.puppala@databricks.com>

---------

Signed-off-by: Vikrant Puppala <vikrant.puppala@databricks.com>

Author: vikrantpuppala

src/databricks/sql/backend/kernel/_errors.py

@@ -38,6 +38,8 @@
 
 from __future__ import annotations
 
+import logging
+
 from databricks.sql.exc import (
     DatabaseError,
     Error,
@@ -57,6 +59,31 @@
         "(into the same venv as databricks-sql-connector)."
     ) from exc
 
+# Route the kernel's Rust-side logs into Python's ``logging`` as soon as
+# the extension loads. The kernel emits under the ``databricks.sql.kernel``
+# logger (a child of the connector's ``databricks.sql`` namespace), so a
+# customer who configures ``databricks.sql`` logging gets kernel logs for
+# free with no extra setup.
+#
+# This is a best-effort, non-essential feature: it must never take down
+# ``use_kernel=True`` for a process. ``getattr`` guards against an older
+# kernel wheel that predates the function. The ``try`` guards against the
+# call itself throwing — note ``except BaseException`` is deliberate: a
+# panic raised across the PyO3 boundary surfaces as
+# ``pyo3_runtime.PanicException``, which derives from ``BaseException``
+# (not ``Exception``), so a narrower clause would let it escape module
+# import and fail every kernel-backed connection. The kernel side is
+# idempotent and returns rather than panics on a double install, but we
+# do not rely on that here — the guard holds regardless of the Rust impl.
+_kernel_init_logging = getattr(_kernel, "init_logging", None)
+if _kernel_init_logging is not None:
+    try:
+        _kernel_init_logging()
+    except BaseException as exc:  # noqa: BLE001 - see comment above re: PanicException
+        logging.getLogger(__name__).debug(
+            "kernel log bridge init failed; continuing without it: %r", exc
+        )
+
 
 # Map a kernel `code` slug to the PEP 249 exception class that best
 # captures it. The match isn't a perfect 1:1 — PEP 249 has a

tests/e2e/test_kernel_backend.py

@@ -127,6 +127,91 @@ def test_fetchall_arrow(conn):
         assert table.column_names == ["a", "b"]
 
 
+# ─── Logging (Rust kernel -> Python logging bridge) ──────────────────────────
+#
+# Layer 3 of the logger-name drift guard (see also the Rust tests
+# `klog::tests::klog_emits_contract_target` and
+# `logging::tests::kernel_target_matches_contract` in the kernel repo).
+# Asserts the *customer-facing* contract end-to-end: kernel logs reach
+# Python `logging` under the `databricks.sql.kernel` logger, respect the
+# level set on it, and the pyo3 boundary surfaces under
+# `databricks.sql.kernel.pyo3`. If the kernel's tracing target or the
+# pyo3-log wiring ever drifts, these fail.
+
+import logging
+
+
+def test_kernel_logs_reach_python_logging(kernel_conn_params, caplog):
+    """A query at DEBUG produces records on the `databricks.sql.kernel`
+    logger — proving the tracing -> log -> pyo3-log -> logging chain."""
+    with caplog.at_level(logging.DEBUG, logger="databricks.sql.kernel"):
+        c = sql.connect(**kernel_conn_params)
+        try:
+            with c.cursor() as cur:
+                cur.execute("SELECT 1 AS a")
+                cur.fetchall()
+        finally:
+            c.close()
+
+    kernel_records = [
+        r for r in caplog.records if r.name.startswith("databricks.sql.kernel")
+    ]
+    assert kernel_records, (
+        "expected log records under the 'databricks.sql.kernel' logger; "
+        "the kernel tracing -> Python logging bridge did not deliver any"
+    )
+    # The core kernel logger (not just any child) must be present — this
+    # is the customer-facing contract.
+    assert any(
+        r.name == "databricks.sql.kernel" for r in kernel_records
+    ), "expected core kernel records on the exact 'databricks.sql.kernel' logger"
+    # The pyo3-boundary breadcrumb (`databricks.sql.kernel.pyo3`) is a
+    # nice-to-have, but the exact sub-target is a kernel-internal naming
+    # detail — assert softly so a benign kernel change to the boundary
+    # breadcrumbs doesn't break the connector e2e suite.
+    if not any(r.name == "databricks.sql.kernel.pyo3" for r in kernel_records):
+        import warnings
+
+        warnings.warn(
+            "no 'databricks.sql.kernel.pyo3' boundary records seen; "
+            "the kernel may have changed its pyo3 breadcrumb target",
+            stacklevel=2,
+        )
+
+
+def test_kernel_log_level_is_respected(kernel_conn_params, caplog):
+    """At WARNING on the kernel logger, no DEBUG/INFO kernel records reach
+    `caplog.records` — i.e. level control on `databricks.sql.kernel`
+    behaves like any other Python logger.
+
+    Scope note: `caplog.at_level` sets the logger's level and attaches a
+    handler, so this asserts the *effective* outcome a customer sees
+    (sub-threshold records don't surface), not specifically that the Rust
+    side suppressed them at source. A DEBUG record that crossed the FFI
+    would still be dropped by Python's level check before reaching
+    `caplog`. Source-side suppression (and its per-record FFI cost
+    avoidance) is covered by the kernel-side filtering, not asserted
+    here."""
+    with caplog.at_level(logging.WARNING, logger="databricks.sql.kernel"):
+        c = sql.connect(**kernel_conn_params)
+        try:
+            with c.cursor() as cur:
+                cur.execute("SELECT 1 AS a")
+                cur.fetchall()
+        finally:
+            c.close()
+
+    debug_records = [
+        r
+        for r in caplog.records
+        if r.name.startswith("databricks.sql.kernel") and r.levelno < logging.WARNING
+    ]
+    assert not debug_records, (
+        "DEBUG/INFO kernel records leaked through at WARNING level: "
+        f"{[(r.name, r.levelname, r.getMessage()) for r in debug_records]}"
+    )
+
+
 # ── Metadata ──────────────────────────────────────────────────────