Repository navigation
Conversation
The SDK's default transport uses HTTP/2 without health-check pings. When a connection dies without a FIN or RST, every request to the host keeps going out over it, including the retries added in #7521, until the OS gives up on TCP (~15 min on Linux). Give the bundle workspace client a transport with HTTP2Config.SendPingTimeout=30s and PingTimeout=15s so the dead connection is closed, and retry "http2: client connection lost" on idempotent calls. Co-authored-by: Isaac <no-reply@databricks.com>
cloud: TestAccept/bundle/invariant/no_drift/DMS=/INPUT_CONFIG=pipeline_allow_duplicate_names.yml.tmpl TestAccept/bundle/invariant/no_drift/DMS=/INPUT_CONFIG=job.yml.tmpl TestAccept/bundle/invariant/delete_idempotent/DMS=/INPUT_CONFIG=pipeline_allow_duplicate_names.yml.tmpl Co-authored-by: Isaac <no-reply@databricks.com>
Collaborator
✅ Integration test reportCommit: 8a92a8c
Top 3 slowest tests (at least 2 minutes):
|
- IsTransient (not only IsTransientRead) retries "http2: client connection lost": the health check now ends a request on a dead connection before the inactivity timeout, which IsTransient already retries. - Set the transport unconditionally; the nil guard could never be false. - Test proxy closes its connections; the health-check test tolerates the connection being closed before the request is sent; test that Workspace.Client picks the transport after insecure_skip_verify is resolved. cloud: TestAccept/bundle/invariant/no_drift/DMS=/INPUT_CONFIG=pipeline_allow_duplicate_names.yml.tmpl TestAccept/bundle/invariant/no_drift/DMS=/INPUT_CONFIG=job.yml.tmpl TestAccept/bundle/invariant/delete_idempotent/DMS=/INPUT_CONFIG=pipeline_allow_duplicate_names.yml.tmpl Co-authored-by: Isaac <no-reply@databricks.com>
denik
marked this pull request as ready for review
November 17, 2026 13:47
denik
requested review from
Sankalp-Mittal,
andrewnester,
anton-107 and
renaudhartert-db
and removed request for
renaudhartert-db
November 17, 2026 13:47
Sankalp-Mittal
left a comment
Contributor
There was a problem hiding this comment.
[AI Generated] Review by review-factory (Claude Opus 5.5 + Claude Fable 5.2, cross-verified). Only points the reviewers agreed on are posted.
andrewnester
left a comment
Contributor
There was a problem hiding this comment.
No blocking findings from a full pass; I'll take a manual look at the remaining changes before approving.
- Note that workspace commands run in a bundle root use the transport too (cmd/root/auth.go builds their client from the bundle client's config). - Name experimental/ssh's newHTTP11Transport in the sync note. - The SDK attribute is skip_verify and has no env var. - Say why isConnectionLost matches by message. Co-authored-by: Isaac <no-reply@databricks.com>
Sankalp-Mittal
left a comment
Contributor
There was a problem hiding this comment.
[AI Generated] Follow-up review by review-factory (Claude Opus 5.5 + Claude Fable 5.2, cross-verified). Only points the reviewers agreed on are posted.
No new issues found.
idelafuente-oc
pushed a commit
to idelafuente-oc/cli
that referenced
this pull request
Nov 20, 2026
…icks#7748) ## Changes `acceptance/bundle/test.toml` gets a new `SuppressLines.warn_dms_transport` pattern. It moves a deployment history warning to the test log when the cause is a transport error: - `stream error: stream ID N; INTERNAL_ERROR; received from peer` - `unexpected EOF` - `context deadline exceeded` (the best-effort drain deadline) It covers the six best-effort warnings: `Failed to record/complete deployment history`, `Could not enable/record/read/delete deployment history`. `bundle/dms/suppress-transport-warnings` pins the pattern with lines it must suppress and lines it must keep. ## Why With `DMS=true`, the bundle suite records deployment history in every test and compares against the same goldens as the plain run. Recording is best effort, so the CLI only warns when a history call fails. When the server resets an HTTP/2 stream mid-response, that extra `Warn:` line fails an otherwise passing test. On 2026-11-17/18 this happened on every cloud, several tests per run. Judgment calls: - **Suppress only transport causes and the drain budget.** `context deadline exceeded` is the `OperationBuffer.Drain` budget, which a slow service also hits; it is best effort too. An error the service returns, such as a validation error or an empty-message 5xx, is still compared. A blanket filter on these warnings would have hidden real CLI bugs like the `operations.resource_key must have a known resource type` regression. The `dms_fault` and `dms` tests assert their injected warnings, which don't match (indented, or `Fault injected by test`). - **No CLI retry.** The reset arrives after the response headers, so the server may already have applied the write. The history writes aren't idempotent: CreateDeployment assigns the id server-side, CreateVersion conflicts on a repeat, and UpdateOperation sends the sequence id from the previous write. `libs/retry.IsTransientRead` only matches `*url.Error`, and these errors come from the SDK's body read (`response body: ...`). databricks#7723 adds HTTP/2 health-check pings for dead connections and does not change this. - **Not covered here:** the same stream reset on resource creates (secrets, model_provider_services) is a real deploy error and stays visible. So do the `[CLI] api get .../resources` failures in test scripts. The backend side is reported to the service owners. ## Tests - A throwaway Go test checked the pattern (as decoded from the TOML) against the failure lines below, plus the lines it must keep (empty-message errors, validation errors, fault-injection lines, resource create errors). - Local `TestAccept/bundle/{dms,dms_fault,resources/secrets,config-remote-sync}` pass. - AWS: `TestAccept/bundle/config-remote-sync/cli_defaults/DMS=true` passes. The commit message runs three affected tests on every cloud. ## Dashboard: | error | snippet | | -- | -- | | [error/3729](http://go/cli/dashboard/error/3729?main=1&pr=1&end=2026-11-18T06:00:00Z&last=3d) | `Warn: Failed to record deployment history: recording operation for resources.pipelines.[NAME]: response body: stream error: stream ID 51; INTERNAL_ERROR; received from peer` | | [error/3731](http://go/cli/dashboard/error/3731?main=1&pr=1&end=2026-11-18T06:00:00Z&last=3d) | `Warn: Failed to complete deployment history version: response body: stream error: stream ID 63; INTERNAL_ERROR; received from peer` | | [error/3732](http://go/cli/dashboard/error/3732?main=1&pr=1&end=2026-11-18T06:00:00Z&last=3d) | `Warn: Failed to record deployment history: recording operation for resources.postgres_roles.[NAME]: response body: stream error: stream ID 69; INTERNAL_ERROR; received from peer` | | [error/3733](http://go/cli/dashboard/error/3733?main=1&pr=1&end=2026-11-18T06:00:00Z&last=3d) | `Warn: Failed to complete deployment history version: response body: stream error: stream ID 9429; INTERNAL_ERROR; received from peer` | | [error/3013](http://go/cli/dashboard/error/3013?main=1&pr=1&end=2026-11-18T06:00:00Z&last=3d) | `Warn: Could not enable deployment history for this deploy: failed to create deployment: response body: stream error: stream ID 43; INTERNAL_ERROR; received from peer` | | [error/3031](http://go/cli/dashboard/error/3031?main=1&pr=1&end=2026-11-18T06:00:00Z&last=3d) | `Warn: Failed to record deployment history: context deadline exceeded` | | [error/3642](http://go/cli/dashboard/error/3642?main=1&pr=1&end=2026-11-18T06:00:00Z&last=3d) | `Warn: Failed to record deployment history: recording operation for resources.catalogs.[NAME]: Patch "[DATABRICKS_URL]/api/2.0/bundle/deployments/[DEPLOYMENT_ID]/versions/<N>/operations/catalogs.catalog1": unexpected EOF` | This pull request and its description were written by Isaac. --------- Co-authored-by: Isaac <no-reply@databricks.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Changes
HTTP2Config.SendPingTimeout = 30s,PingTimeout = 15s(bundle/config/transport.go). Otherwise it matches the SDK's default transport and carries overskip_verify. It is set inWorkspace.Clientafter the config is resolved.libs/retry.IsTransientalso retrieshttp2: client connection lost, the error a request gets when the health check closes its connection. It replaces the inactivity timeout for such requests, whichIsTransientalready retries (#7521), with the same uncertainty about whether the server applied the request. Non-overwrite writes such as the deploy-lock create are still not retried.Why
#7521 retries the SDK's "request timed out after 1m30s of inactivity". Since it landed, error/179 has failed twice on main (11-13 Azure linux, 11-17 GCP windows), and both runs show the same pattern:
import-file?overwrite=trueupload of the deploy retried twice and timed out every time, followed by the deploy-lockget-status;read tcp ...: wsarecv: A connection attempt failed because the connected party did not properly respond after a period of time, which is the OS giving up on the TCP connection.So the workspace was fine; one process was stuck on a dead connection. The SDK transport has
ForceAttemptHTTP2and no health check, so all requests to the host share one TCP connection. A request that times out only resets its stream. Go then keeps using the connection while it has free stream slots (the ping sent with the reset has no timeout), so the retries go out over the same dead socket until the OS closes it (~15 min with Linux defaults). With the health check, a dead connection is closed ≤45s after it goes silent, the in-flight requests fail withhttp2: client connection lost, and the retry dials a new connection.Judgment calls:
cmd/root/auth.gobuilds their client from the bundle client's config. Other commands, and the token-refresh client the SDK builds while resolving the config, still use the SDK default transport. The long-term home is the SDK's default transport; this PR is the minimal fix for the CI failure.httpclient.makeDefaultTransport) field by field. It needs to be kept in sync on SDK upgrades, and the code comment says so.experimental/ssh/internal/client/releases.go(newHTTP11Transport) is another such copy, HTTP/1.1-only, and already lacksMaxIdleConnsPerHost; the comment names it, and sharing one base is left for a follow-up so this PR doesn't change the ssh client.Tests
TestTransportWithoutHealthCheckReusesDeadConnectionreproduces the failure: an HTTP/2 server behind a TCP proxy that silently drops an established connection; without the health check, a second request after a timeout goes out on the same dead connection.TestHealthCheckedTransportReplacesDeadConnection: with the health check, the request fails withhttp2: client connection lostand the next one dials a new connection.TestWorkspaceClientUsesHealthCheckedTransport:Workspace.Clientpicks the transport afterskip_verifyfrom the profile is resolved.IsTransientmessage. Local acceptanceTestAccept/bundleandTestAccept/authpass. On AWS,bundle/invariantno_driftanddelete_idempotentpass forjobandpipeline_allow_duplicate_names(DMS on and off). The last commit runs them on every cloud.Dashboard:
Error: Post "[DATABRICKS_URL]/api/2.0/workspace-files/import-file/[PATH]": request timed out after 1m30s of inactivityThe 11-17 failure (TestAccept/bundle/invariant/no_drift/DMS=/INPUT_CONFIG=job_with_task.yml.tmpl/READPLAN=1) isn't ingested by the dashboard yet; snippet from the job log:
This pull request and its description were written by Isaac.