Skip to content

Use the health-checked HTTP/2 transport in the test harness clients - #7797

Open
denik wants to merge 3 commits into
denik/http2-health-checkfrom
denik/http2-health-check-harness
Open

denik wants to merge 3 commits into
denik/http2-health-checkfrom
denik/http2-health-check-harness

Conversation

@denik

@denik denik commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

Changes

Stacked on #7723.

  • Move the health-checked HTTP/2 transport from bundle/config to a new libs/httptransport package and export it as httptransport.HealthChecked(insecureSkipVerify). The code is unchanged; the Workspace.Client test stays in bundle/config.
  • Use it in the two test-harness clients that live for the whole test process:
    • integration/internal/acc: the shared workspace client of the integration tests (wt.W).
    • libs/testproxy: the recording proxy that acceptance tests put in front of the real workspace.

Why

These request timeouts are the same dead-connection failure that #7723 fixes, in the test harness instead of the CLI. The SDK uses one process-wide default transport. Both clients above share it for every test in the go test process, so when the runner's connection to the workspace is silently dropped, every later request goes out over the same dead connection. Each one then times out after the 60s inactivity limit:

  • In the integration jobs, consecutive tests fail exactly one minute apart until the package runs out of tests. For example, 7 integration/cmd/sync tests from 02:29 to 02:35 in one Azure job, and 8 from 17:57 to 18:04 in one GCP Windows job.
  • In acceptance, the error arrives through the proxy as a 500 whose message is the proxy's own timeout (the 1m0s ... (500 UNKNOWN) below). The CLI's own client has a 1m30s timeout.

#7723 covers only the CLI's bundle client, which issues the 1m30s variants. With this PR, a dead connection in the test harness is closed within 45s and the next request dials a new one. At most the in-flight request fails, not every test that follows.

Judgment calls:

  • Moved the transport to libs/ rather than importing bundle/config from the test harness.
  • Both clients set the transport before their config is resolved, so the SDK's token-refresh client uses it too. That client lives as long as the shared client does. With a transport set, the SDK ignores skip_verify; the integration tests and the proxy don't use it.
  • Left as is: the acceptance runner's cloudWorkspace client, which makes two calls once per process, and the CLI's non-bundle commands (production behavior; the SDK default transport is the long-term home for the health check).
  • No changelog entry: no user-visible change beyond Close dead HTTP/2 connections in the bundle workspace client #7723.

Tests

  • Moved unit tests pass (libs/httptransport, bundle/config), as does lint. New libs/testproxy tests check the proxy's transport.
  • Local acceptance: bundle/resources/permissions/jobs/delete_one, selftest, bundle/deploy/files.
  • Ran on AWS with testme: acceptance bundle/resources/permissions/jobs/delete_one (goes through the proxy) and integration cmd/sync TestSyncEnsureRemotePathIsUsable*. The last commit runs the acceptance test on AWS in CI.
  • The dead-connection condition itself cannot be triggered on demand in CI; libs/httptransport tests reproduce it with a blackhole proxy.

Dashboard:

error snippet
error/1091 Received unexpected error: Get "[DATABRICKS_URL]/api/2.0/preview/scim/v2/Me": request timed out after 1m0s of inactivity
error/2877 Received unexpected error: Post "[DATABRICKS_URL]/api/2.0/dbfs/mkdirs": request timed out after 1m0s of inactivity
error/2950 Error: Post "[DATABRICKS_URL]/api/2.0/workspace-files/import-file/[PATH]": request timed out after 1m0s of inactivity (500 UNKNOWN)

This pull request and its description were written by Isaac.

Move the health-checked transport from bundle/config to libs/httptransport
and use it in the shared integration-test workspace client and in the
acceptance test proxy. Both live for the whole test process and share one
connection pool, so a silently dropped connection made every later test
in the job hang until the inactivity timeout.

aws: TestAccept/bundle/resources/permissions/jobs/delete_one

Co-authored-by: Isaac <no-reply@databricks.com>
@denik denik added the ai-auto Deprecated: use factory-generated instead. label Nov 19, 2026
@denik
denik added this pull request to stack #7798 November 19, 2026 10:01
Also keep a caller's transport in the test proxy, fix a stale pointer in
libs/retry and add tests for the proxy transport.

aws: TestAccept/bundle/resources/permissions/jobs/delete_one

Co-authored-by: Isaac <no-reply@databricks.com>
@github-actions github-actions Bot added the DABs DABs related issues label Nov 19, 2026
@eng-dev-ecosystem-bot

eng-dev-ecosystem-bot commented Nov 19, 2026 •

Copy link
Copy Markdown
Collaborator

✅ Integration test report

Commit: d17f5e5

Run: 44512840187

Env 🤯​MISS 💚​RECOVERED ✅​pass 🙈​skip Time
💚​ aws linux-2core-8gb 1 275 19 6:34
💚​ aws-windows-latest-4core-16gb 1 277 17 4:42
🤯​ azure linux-2core-8gb 1 1 271 19 5:12
🤯​ azure-windows-latest-4core-16gb 1 1 273 17 4:08
🤯​ gcp linux-2core-8gb 1 1 272 19 6:17
🤯​ gcp-windows-latest-4core-16gb 1 1 274 17 3:58
Test Name aws linux-2core-8gb aws-windows-latest-4core-16gb azure linux-2core-8gb azure-windows-latest-4core-16gb gcp linux-2core-8gb gcp-windows-latest-4core-16gb
💚​ TestAccept 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R
🤯​ TestAccept/bundle/resources/permissions/jobs/delete_one ✅​p ✅​p 🤯​M 🤯​M 🤯​M 🤯​M
Top 3 slowest tests (at least 2 minutes):
duration env testname
3:52 azure-windows-latest-4core-16gb TestAccept
3:28 aws-windows-latest-4core-16gb TestAccept
3:15 gcp-windows-latest-4core-16gb TestAccept

@denik
denik marked this pull request as ready for review November 19, 2026 10:41
@denik
denik requested review from a team as code owners November 19, 2026 10:41
@denik
denik requested review from Sankalp-Mittal, andrewnester and parthban-db and removed request for parthban-db November 19, 2026 10:41

@review-oss-bot review-oss-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me — no issues found.

@Sankalp-Mittal Sankalp-Mittal left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[AI Generated] Review by review-factory (Claude Opus 5.5 + Claude Fable 5.2, cross-verified). Only points the reviewers agreed on are posted.

Comment thread libs/httptransport/transport.go Resolved
Comment thread libs/testproxy/server.go Resolved
…proxy transport

aws: TestAccept/bundle/resources/permissions/jobs/delete_one

Co-authored-by: Isaac <no-reply@databricks.com>

@Sankalp-Mittal Sankalp-Mittal left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[AI Generated] Follow-up review by review-factory (Claude Opus 5.5 + Claude Fable 5.2, cross-verified). Only points the reviewers agreed on are posted.

No new issues found.

@denik denik added factory-generated and removed ai-auto Deprecated: use factory-generated instead. labels Nov 20, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

DABs DABs related issues factory-generated

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants