ci(sea): Kernel E2E workflow — build napi binding from KERNEL_REV + run tests/e2e/sea on a live warehouse#418
Open
msrathore-db wants to merge 4 commits into
Open
ci(sea): Kernel E2E workflow — build napi binding from KERNEL_REV + run tests/e2e/sea on a live warehouse#418msrathore-db wants to merge 4 commits into
msrathore-db wants to merge 4 commits into
Conversation
…, run tests/e2e/sea on a live warehouse
Brings the SEA-backend end-to-end gate onto main (it previously lived only in the
stacked sea-kernel-rev-pin branch). Mirrors the Python connector's kernel-e2e.yml.
What it does:
- Reads the pinned kernel SHA from KERNEL_REV, checks the kernel out via a GitHub
App token (private repo), runs `npm run build:native` to compile the napi
binding into native/sea/ — so the binary matches KERNEL_REV exactly.
- Asserts the committed native/sea/index.{d.ts,js} contract matches the rebuilt
binding (catches drift between the pin and the committed router/types).
- Smoke-checks the binding loads, then runs `tests/e2e/sea/**` against the
azure-prod warehouse (maps the secrets onto the DATABRICKS_PECOTESTING_* vars
the SEA suite gates on).
Gate semantics (same pattern as the Thrift integration + Python kernel-e2e):
- Plain PR events post a synthetic-success "Kernel E2E" check so PRs that don't
touch SEA aren't blocked; the real suite runs in the merge queue.
- The `kernel-e2e` label triggers a preview run on the PR and is auto-removed on
push (a labelled run can't be re-triggered with unreviewed code).
- merge_group runs the real gate when SEA-relevant files changed, auto-passes
otherwise.
All referenced pieces already exist on main: the setup-jfrog composite (npm + the
inline cargo proxy config), KERNEL_REV, the build:native script, the committed
native/sea contract, and tests/e2e/sea/.
One-time admin setup required before this gates anything (documented in the
workflow header): the `kernel-e2e` label, the INTEGRATION_TEST_APP_ID /
INTEGRATION_TEST_PRIVATE_KEY GitHub App (allowlisted for databricks-sql-kernel),
and the azure-prod environment exposing DATABRICKS_HOST /
TEST_PECO_WAREHOUSE_HTTP_PATH / DATABRICKS_TOKEN.
Co-authored-by: Isaac
Signed-off-by: Madhavendra Rathore <madhavendra.rathore@databricks.com>
…process.exit fix) The tests/e2e/sea suite reads creds from two sources: most tests read DATABRICKS_PECOTESTING_* directly (already injected), but e2e-smoke.test.ts imports tests/e2e/utils/config.ts, which process.exit(1)s at module load if any E2E_* var is unset — aborting the whole mocha run before it starts. Mirror main.yml's e2e-test job mapping (E2E_HOST/PATH/ACCESS_TOKEN + CATALOG/SCHEMA/ VOLUME/TABLE_SUFFIX) so the suite loads and runs. Co-authored-by: Isaac Signed-off-by: Madhavendra Rathore <madhavendra.rathore@databricks.com>
msrathore-db
added
kernel-e2e
…'t hit unreachable dev proxy
Two package-lock entries (flatbuffers etc.) resolve to npm-proxy.dev.databricks.com,
which the protected runner can't reach (ECONNRESET). main.yml's e2e job only
succeeds because it caches ~/.npm — those tarballs load from the warm cache, never
fetched. kernel-e2e had no such cache, so npm ci failed deterministically.
Add the identical actions/cache step (same key) before npm ci. The driver has a
single root package-lock.json and the kernel checkout adds none, so
hashFiles('**/package-lock.json') matches main.yml's key exactly → restores main's
warmed cache. Confirmed end-to-end up to this point: App token + private-kernel
checkout + JFrog now all pass; npm ci was the only remaining failure.
Co-authored-by: Isaac
Signed-off-by: Madhavendra Rathore <madhavendra.rathore@databricks.com>
github-actions
Bot
removed
the
kernel-e2e
…-loader race) Bare `mocha` has no ts-node registration, so it loaded the .ts specs via the ESM dynamic-import path and crashed with ERR_INTERNAL_ASSERTION (... not yet fully loaded ... Promise.all) before any test ran. main.yml's e2e job avoids this by running `nyc mocha` — nyc.config.js's `require: ['ts-node/register']` installs the CommonJS ts-node hook, loading specs synchronously. Mirror that: invoke `nyc ... mocha --config tests/e2e/.mocharc.js <glob>`, keeping the glob at argv index 4 (the mocharc derives spec from process.argv.slice(4)). Everything else in the pipeline is now confirmed green: App-token + private-kernel checkout, ~/.npm cache, npm ci, build:native, the KERNEL_REV contract-drift assert, and the binding smoke-check. This was the last failing step. Co-authored-by: Isaac Signed-off-by: Madhavendra Rathore <madhavendra.rathore@databricks.com>
github-actions
Bot
removed
the
kernel-e2e
msrathore-db
removed
the
kernel-e2e
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds
.github/workflows/kernel-e2e.ymlto main — the SEA-backend end-to-end gate. It previously lived only in the strandedsea-kernel-rev-pinstack; this lands it independently offmain. It mirrors the Python connector'skernel-e2e.yml.How it works
KERNEL_REV, checks the kernel out via a GitHub App token (private repo), and runsnpm run build:nativeto compile the napi binding intonative/sea/— so the.nodematchesKERNEL_REVexactly.native/sea/index.{d.ts,js}contract matches the rebuilt binding (catches drift between the pin and the committed router/types).tests/e2e/sea/**against theazure-prodwarehouse (maps the secrets onto theDATABRICKS_PECOTESTING_*vars the SEA suite gates on).Gate semantics (same pattern as Thrift integration + Python kernel-e2e)
Kernel E2Echeck is posted so PRs that don't touch SEA aren't blocked. The real suite runs in the merge queue.kernel-e2elabel: triggers a preview run on the PR; auto-removed on push (a labelled run can't be re-triggered with unreviewed code).lib/sea/,native/sea/,tests/e2e/sea/,tests/unit/sea/,KERNEL_REV,package*.json, this workflow), auto-passes otherwise.Self-contained on main
All referenced pieces already exist on
main: thesetup-jfrogcomposite (npm proxy + the inline cargo proxy config),KERNEL_REV, thebuild:nativescript, the committednative/seacontract, andtests/e2e/sea/. YAML validated.Documented in the workflow header:
kernel-e2elabel.INTEGRATION_TEST_APP_ID/INTEGRATION_TEST_PRIVATE_KEYGitHub App secrets, App allowlisted fordatabricks/databricks-sql-kernel.azure-prodenvironment exposingDATABRICKS_HOST/TEST_PECO_WAREHOUSE_HTTP_PATH/DATABRICKS_TOKEN.Kernel E2Eto the merge-queue required checks once verified.Until then, the workflow posts synthetic-success on PRs (non-blocking) and is a no-op without the label/secrets — safe to merge.
This pull request and its description were written by Isaac.