Skip to content

Close dead HTTP/2 connections in the bundle workspace client - #7723

Open
denik wants to merge 4 commits into
mainfrom
denik/http2-health-check
Open

denik wants to merge 4 commits into
mainfrom
denik/http2-health-check

Conversation

@denik

@denik denik commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

Changes

  • The bundle workspace client now uses its own HTTP/2 transport with health-check pings: HTTP2Config.SendPingTimeout = 30s, PingTimeout = 15s (bundle/config/transport.go). Otherwise it matches the SDK's default transport and carries over skip_verify. It is set in Workspace.Client after the config is resolved.
  • libs/retry.IsTransient also retries http2: client connection lost, the error a request gets when the health check closes its connection. It replaces the inactivity timeout for such requests, which IsTransient already retries (#7521), with the same uncertainty about whether the server applied the request. Non-overwrite writes such as the deploy-lock create are still not retried.

Why

#7521 retries the SDK's "request timed out after 1m30s of inactivity". Since it landed, error/179 has failed twice on main (11-13 Azure linux, 11-17 GCP windows), and both runs show the same pattern:

  • every import-file?overwrite=true upload of the deploy retried twice and timed out every time, followed by the deploy-lock get-status;
  • the test hung for 631s (GCP windows) and 909s (Azure linux), while ~50 (resp. 45) other tests in the same job passed against the same workspace in that window;
  • on Windows the last attempt failed with read tcp ...: wsarecv: A connection attempt failed because the connected party did not properly respond after a period of time, which is the OS giving up on the TCP connection.

So the workspace was fine; one process was stuck on a dead connection. The SDK transport has ForceAttemptHTTP2 and no health check, so all requests to the host share one TCP connection. A request that times out only resets its stream. Go then keeps using the connection while it has free stream slots (the ping sent with the reset has no timeout), so the retries go out over the same dead socket until the OS closes it (~15 min with Linux defaults). With the health check, a dead connection is closed ≤45s after it goes silent, the in-flight requests fail with http2: client connection lost, and the retry dials a new connection.

Judgment calls:

  • Bundle workspace client only. That's where we see it and where the retries live. Workspace commands run in a bundle root use it too, because cmd/root/auth.go builds their client from the bundle client's config. Other commands, and the token-refresh client the SDK builds while resolving the config, still use the SDK default transport. The long-term home is the SDK's default transport; this PR is the minimal fix for the CI failure.
  • The transport copies the SDK's unexported default (httpclient.makeDefaultTransport) field by field. It needs to be kept in sync on SDK upgrades, and the code comment says so. experimental/ssh/internal/client/releases.go (newHTTP11Transport) is another such copy, HTTP/1.1-only, and already lacks MaxIdleConnsPerHost; the comment names it, and sharing one base is left for a follow-up so this PR doesn't change the ssh client.
  • 30s/15s are the values Kubernetes' client-go uses. The ping cost is one frame per 30s on a silent connection.

Tests

  • TestTransportWithoutHealthCheckReusesDeadConnection reproduces the failure: an HTTP/2 server behind a TCP proxy that silently drops an established connection; without the health check, a second request after a timeout goes out on the same dead connection.
  • TestHealthCheckedTransportReplacesDeadConnection: with the health check, the request fails with http2: client connection lost and the next one dials a new connection.
  • TestWorkspaceClientUsesHealthCheckedTransport: Workspace.Client picks the transport after skip_verify from the profile is resolved.
  • Unit tests for the new IsTransient message. Local acceptance TestAccept/bundle and TestAccept/auth pass. On AWS, bundle/invariant no_drift and delete_idempotent pass for job and pipeline_allow_duplicate_names (DMS on and off). The last commit runs them on every cloud.

Dashboard:

error snippet
error/179 Error: Post "[DATABRICKS_URL]/api/2.0/workspace-files/import-file/[PATH]": request timed out after 1m30s of inactivity

The 11-17 failure (TestAccept/bundle/invariant/no_drift/DMS=/INPUT_CONFIG=job_with_task.yml.tmpl/READPLAN=1) isn't ingested by the dashboard yet; snippet from the job log:

Error: Post "[DATABRICKS_URL]/api/2.0/workspace-files/import-file/...dashboard.lvdash.json?overwrite=true": request timed out after 1m30s of inactivity
Error: unlock called when lock is not held: Get "[DATABRICKS_URL]/api/2.0/workspace/get-status?path=...deploy.lock&return_export_info=true": read tcp [IP]->[IP]: wsarecv: A connection attempt failed because the connected party did not properly respond after a period of time, ...

This pull request and its description were written by Isaac.

The SDK's default transport uses HTTP/2 without health-check pings. When a
connection dies without a FIN or RST, every request to the host keeps going
out over it, including the retries added in #7521, until the OS gives up on
TCP (~15 min on Linux). Give the bundle workspace client a transport with
HTTP2Config.SendPingTimeout=30s and PingTimeout=15s so the dead connection is
closed, and retry "http2: client connection lost" on idempotent calls.

Co-authored-by: Isaac <no-reply@databricks.com>
@denik denik added the ai-auto Deprecated: use factory-generated instead. label Nov 17, 2026
cloud: TestAccept/bundle/invariant/no_drift/DMS=/INPUT_CONFIG=pipeline_allow_duplicate_names.yml.tmpl TestAccept/bundle/invariant/no_drift/DMS=/INPUT_CONFIG=job.yml.tmpl TestAccept/bundle/invariant/delete_idempotent/DMS=/INPUT_CONFIG=pipeline_allow_duplicate_names.yml.tmpl

Co-authored-by: Isaac <no-reply@databricks.com>
@github-actions github-actions Bot added the DABs DABs related issues label Nov 17, 2026
@eng-dev-ecosystem-bot

eng-dev-ecosystem-bot commented Nov 17, 2026 •

Copy link
Copy Markdown
Collaborator

✅ Integration test report

Commit: 8a92a8c

Run: 44250932359

Env 💚​RECOVERED ✅​pass 🙈​skip Time
💚​ aws linux-2core-8gb 1 272 19 4:20
💚​ aws-windows-latest-4core-16gb 1 274 17 4:00
💚​ azure linux-2core-8gb 1 271 19 3:51
💚​ azure-windows-latest-4core-16gb 1 273 17 3:53
💚​ gcp linux-2core-8gb 1 272 19 4:02
💚​ gcp-windows-latest-4core-16gb 1 274 17 4:02
Test Name aws linux-2core-8gb aws-windows-latest-4core-16gb azure linux-2core-8gb azure-windows-latest-4core-16gb gcp linux-2core-8gb gcp-windows-latest-4core-16gb
💚​ TestAccept 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R
Top 3 slowest tests (at least 2 minutes):
duration env testname
3:57 gcp-windows-latest-4core-16gb TestAccept
3:55 aws-windows-latest-4core-16gb TestAccept
3:48 azure-windows-latest-4core-16gb TestAccept

- IsTransient (not only IsTransientRead) retries "http2: client connection
  lost": the health check now ends a request on a dead connection before the
  inactivity timeout, which IsTransient already retries.
- Set the transport unconditionally; the nil guard could never be false.
- Test proxy closes its connections; the health-check test tolerates the
  connection being closed before the request is sent; test that
  Workspace.Client picks the transport after insecure_skip_verify is resolved.

cloud: TestAccept/bundle/invariant/no_drift/DMS=/INPUT_CONFIG=pipeline_allow_duplicate_names.yml.tmpl TestAccept/bundle/invariant/no_drift/DMS=/INPUT_CONFIG=job.yml.tmpl TestAccept/bundle/invariant/delete_idempotent/DMS=/INPUT_CONFIG=pipeline_allow_duplicate_names.yml.tmpl

Co-authored-by: Isaac <no-reply@databricks.com>
@denik
denik marked this pull request as ready for review November 17, 2026 13:47
@denik
denik requested review from a team as code owners November 17, 2026 13:47

@Sankalp-Mittal Sankalp-Mittal left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[AI Generated] Review by review-factory (Claude Opus 5.5 + Claude Fable 5.2, cross-verified). Only points the reviewers agreed on are posted.

Comment thread bundle/config/transport.go Resolved
Comment thread bundle/config/transport.go Resolved
Comment thread bundle/config/workspace.go Resolved
Comment thread libs/retry/retry.go Resolved

@andrewnester andrewnester left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No blocking findings from a full pass; I'll take a manual look at the remaining changes before approving.

- Note that workspace commands run in a bundle root use the transport too
  (cmd/root/auth.go builds their client from the bundle client's config).
- Name experimental/ssh's newHTTP11Transport in the sync note.
- The SDK attribute is skip_verify and has no env var.
- Say why isConnectionLost matches by message.

Co-authored-by: Isaac <no-reply@databricks.com>

@Sankalp-Mittal Sankalp-Mittal left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[AI Generated] Follow-up review by review-factory (Claude Opus 5.5 + Claude Fable 5.2, cross-verified). Only points the reviewers agreed on are posted.

No new issues found.

@denik
denik added this pull request to stack #7798 November 19, 2026 10:01
@denik denik added factory-generated and removed ai-auto Deprecated: use factory-generated instead. labels Nov 20, 2026
idelafuente-oc pushed a commit to idelafuente-oc/cli that referenced this pull request Nov 20, 2026
…icks#7748)

## Changes
`acceptance/bundle/test.toml` gets a new
`SuppressLines.warn_dms_transport` pattern. It moves a deployment
history warning to the test log when the cause is a transport error:
- `stream error: stream ID N; INTERNAL_ERROR; received from peer`
- `unexpected EOF`
- `context deadline exceeded` (the best-effort drain deadline)

It covers the six best-effort warnings: `Failed to record/complete
deployment history`, `Could not enable/record/read/delete deployment
history`. `bundle/dms/suppress-transport-warnings` pins the pattern with
lines it must suppress and lines it must keep.

## Why
With `DMS=true`, the bundle suite records deployment history in every
test and compares against the same goldens as the plain run. Recording
is best effort, so the CLI only warns when a history call fails. When
the server resets an HTTP/2 stream mid-response, that extra `Warn:` line
fails an otherwise passing test. On 2026-11-17/18 this happened on every
cloud, several tests per run.

Judgment calls:
- **Suppress only transport causes and the drain budget.** `context
deadline exceeded` is the `OperationBuffer.Drain` budget, which a slow
service also hits; it is best effort too. An error the service returns,
such as a validation error or an empty-message 5xx, is still compared. A
blanket filter on these warnings would have hidden real CLI bugs like
the `operations.resource_key must have a known resource type`
regression. The `dms_fault` and `dms` tests assert their injected
warnings, which don't match (indented, or `Fault injected by test`).
- **No CLI retry.** The reset arrives after the response headers, so the
server may already have applied the write. The history writes aren't
idempotent: CreateDeployment assigns the id server-side, CreateVersion
conflicts on a repeat, and UpdateOperation sends the sequence id from
the previous write. `libs/retry.IsTransientRead` only matches
`*url.Error`, and these errors come from the SDK's body read (`response
body: ...`). databricks#7723 adds HTTP/2 health-check pings for dead connections
and does not change this.
- **Not covered here:** the same stream reset on resource creates
(secrets, model_provider_services) is a real deploy error and stays
visible. So do the `[CLI] api get .../resources` failures in test
scripts. The backend side is reported to the service owners.

## Tests
- A throwaway Go test checked the pattern (as decoded from the TOML)
against the failure lines below, plus the lines it must keep
(empty-message errors, validation errors, fault-injection lines,
resource create errors).
- Local
`TestAccept/bundle/{dms,dms_fault,resources/secrets,config-remote-sync}`
pass.
- AWS: `TestAccept/bundle/config-remote-sync/cli_defaults/DMS=true`
passes. The commit message runs three affected tests on every cloud.

## Dashboard:
| error | snippet |
| -- | -- |
|
[error/3729](http://go/cli/dashboard/error/3729?main=1&pr=1&end=2026-11-18T06:00:00Z&last=3d)
| `Warn: Failed to record deployment history: recording operation for
resources.pipelines.[NAME]: response body: stream error: stream ID 51;
INTERNAL_ERROR; received from peer` |
|
[error/3731](http://go/cli/dashboard/error/3731?main=1&pr=1&end=2026-11-18T06:00:00Z&last=3d)
| `Warn: Failed to complete deployment history version: response body:
stream error: stream ID 63; INTERNAL_ERROR; received from peer` |
|
[error/3732](http://go/cli/dashboard/error/3732?main=1&pr=1&end=2026-11-18T06:00:00Z&last=3d)
| `Warn: Failed to record deployment history: recording operation for
resources.postgres_roles.[NAME]: response body: stream error: stream ID
69; INTERNAL_ERROR; received from peer` |
|
[error/3733](http://go/cli/dashboard/error/3733?main=1&pr=1&end=2026-11-18T06:00:00Z&last=3d)
| `Warn: Failed to complete deployment history version: response body:
stream error: stream ID 9429; INTERNAL_ERROR; received from peer` |
|
[error/3013](http://go/cli/dashboard/error/3013?main=1&pr=1&end=2026-11-18T06:00:00Z&last=3d)
| `Warn: Could not enable deployment history for this deploy: failed to
create deployment: response body: stream error: stream ID 43;
INTERNAL_ERROR; received from peer` |
|
[error/3031](http://go/cli/dashboard/error/3031?main=1&pr=1&end=2026-11-18T06:00:00Z&last=3d)
| `Warn: Failed to record deployment history: context deadline exceeded`
|
|
[error/3642](http://go/cli/dashboard/error/3642?main=1&pr=1&end=2026-11-18T06:00:00Z&last=3d)
| `Warn: Failed to record deployment history: recording operation for
resources.catalogs.[NAME]: Patch
"[DATABRICKS_URL]/api/2.0/bundle/deployments/[DEPLOYMENT_ID]/versions/<N>/operations/catalogs.catalog1":
unexpected EOF` |

This pull request and its description were written by Isaac.

---------

Co-authored-by: Isaac <no-reply@databricks.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

DABs DABs related issues factory-generated

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants