Repository navigation
Close dead HTTP/2 connections in the bundle workspace client #7723
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
+315
−3
Open
Changes from all commits
Commits
Show all changes
4 commits
Select commit
Hold shift + click to select a range
482310d
Close dead HTTP/2 connections in the bundle workspace client
denik 7c297f1
Add changelog entry
denik a7d3bed
Address review: retry connection-lost on all calls, harden tests
denik 8a92a8c
Address review: document transport scope, sync note and error matching
denik File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1 @@ | ||
| * Close dead HTTP/2 connections during bundle commands, so retries after a request timeout go out on a new connection instead of hanging until the OS drops the old one. ([#7723](https://github.com/databricks/cli/pull/7723)) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,68 @@ | ||
| package config | ||
|
|
||
| import ( | ||
| "crypto/tls" | ||
| "net" | ||
| "net/http" | ||
| "runtime" | ||
| "sync" | ||
| "time" | ||
| ) | ||
|
|
||
| // http2SendPingTimeout and http2PingTimeout make the HTTP/2 client ping a connection that has been | ||
| // silent for 30s and close it if the ping is not answered within 15s. | ||
| // | ||
| // Without them, a connection that dies without a FIN or RST (seen in CI as a single process's | ||
| // requests all hanging while other processes reach the same workspace fine) stays in the pool until | ||
| // the OS gives up on TCP, which takes ~15 minutes on Linux. A request that times out only resets its | ||
| // own stream, so the retries in libs/retry go out over the same dead connection and time out too. | ||
| const ( | ||
| http2SendPingTimeout = 30 * time.Second | ||
| http2PingTimeout = 15 * time.Second | ||
| ) | ||
|
|
||
| // newHealthCheckedTransport mirrors the SDK's default transport (httpclient.makeDefaultTransport, | ||
| // which is unexported) and adds the HTTP/2 health check. Keep the two in sync on SDK upgrades. | ||
| // experimental/ssh/internal/client/releases.go (newHTTP11Transport) is another hand copy of the | ||
| // same defaults, HTTP/1.1-only; check it too. | ||
| // | ||
| // The bundle workspace client uses it, and so do workspace commands run in a bundle root: | ||
| // cmd/root/auth.go builds their client from the bundle client's config. The token-refresh client | ||
| // the SDK builds while resolving the config still uses the SDK default. | ||
| func newHealthCheckedTransport(insecureSkipVerify bool) *http.Transport { | ||
| t := &http.Transport{ | ||
| Proxy: http.ProxyFromEnvironment, | ||
| DialContext: (&net.Dialer{ | ||
| Timeout: 30 * time.Second, | ||
| KeepAlive: 30 * time.Second, | ||
| }).DialContext, | ||
| ForceAttemptHTTP2: true, | ||
| MaxIdleConns: 100, | ||
| MaxIdleConnsPerHost: runtime.GOMAXPROCS(0) + 1, | ||
| IdleConnTimeout: 180 * time.Second, | ||
| TLSHandshakeTimeout: 30 * time.Second, | ||
| ExpectContinueTimeout: 1 * time.Second, | ||
| HTTP2: &http.HTTP2Config{ | ||
| SendPingTimeout: http2SendPingTimeout, | ||
| PingTimeout: http2PingTimeout, | ||
| }, | ||
| } | ||
| if insecureSkipVerify { | ||
| // The SDK only applies InsecureSkipVerify to its own transport, so we carry it over. | ||
| t.TLSClientConfig = &tls.Config{InsecureSkipVerify: true} //nolint:gosec // the user opted in via skip_verify | ||
| } | ||
| return t | ||
| } | ||
|
|
||
| // The transports are shared, like the SDK's default one, so clients reuse pooled connections. | ||
| var ( | ||
| healthCheckedTransport = sync.OnceValue(func() *http.Transport { return newHealthCheckedTransport(false) }) | ||
| insecureHealthCheckedTransport = sync.OnceValue(func() *http.Transport { return newHealthCheckedTransport(true) }) | ||
| ) | ||
|
|
||
| func workspaceTransport(insecureSkipVerify bool) *http.Transport { | ||
| if insecureSkipVerify { | ||
| return insecureHealthCheckedTransport() | ||
| } | ||
| return healthCheckedTransport() | ||
| } | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,219 @@ | ||
| package config | ||
|
|
||
| import ( | ||
| "context" | ||
| "io" | ||
| "net" | ||
| "net/http" | ||
| "net/http/httptest" | ||
| "sync" | ||
| "sync/atomic" | ||
| "testing" | ||
| "time" | ||
|
|
||
| "github.com/databricks/cli/libs/databrickscfg" | ||
| "github.com/databricks/cli/libs/retry" | ||
| "github.com/databricks/databricks-sdk-go/config" | ||
| "github.com/stretchr/testify/assert" | ||
| "github.com/stretchr/testify/require" | ||
| ) | ||
|
|
||
| // blackholeProxy forwards TCP connections to target until blackhole is called. After that, the | ||
| // connections it already accepted stop forwarding bytes in both directions but stay open, like a | ||
| // connection silently dropped by a NAT or load balancer. New connections still work. | ||
| type blackholeProxy struct { | ||
| ln net.Listener | ||
| target string | ||
| mu sync.Mutex | ||
| links []*proxyLink | ||
| accepted atomic.Int32 | ||
| } | ||
|
|
||
| type proxyLink struct { | ||
| client, server net.Conn | ||
| dead atomic.Bool | ||
| } | ||
|
|
||
| func newBlackholeProxy(t *testing.T, target string) *blackholeProxy { | ||
| ln, err := net.Listen("tcp", "127.0.0.1:0") | ||
| require.NoError(t, err) | ||
| p := &blackholeProxy{ln: ln, target: target} | ||
| t.Cleanup(p.close) | ||
| go p.serve() | ||
| return p | ||
| } | ||
|
|
||
| func (p *blackholeProxy) serve() { | ||
| for { | ||
| client, err := p.ln.Accept() | ||
| if err != nil { | ||
| return | ||
| } | ||
| server, err := net.Dial("tcp", p.target) | ||
| if err != nil { | ||
| client.Close() | ||
| continue | ||
| } | ||
| p.accepted.Add(1) | ||
| l := &proxyLink{client: client, server: server} | ||
| p.mu.Lock() | ||
| p.links = append(p.links, l) | ||
| p.mu.Unlock() | ||
| go l.pipe(server, client) | ||
| go l.pipe(client, server) | ||
| } | ||
| } | ||
|
|
||
| func (l *proxyLink) pipe(dst, src net.Conn) { | ||
| defer l.close() | ||
| buf := make([]byte, 32*1024) | ||
| for { | ||
| n, err := src.Read(buf) | ||
| if err != nil { | ||
| return | ||
| } | ||
| if l.dead.Load() { | ||
| continue | ||
| } | ||
| if _, err := dst.Write(buf[:n]); err != nil { | ||
| return | ||
| } | ||
| } | ||
| } | ||
|
|
||
| func (l *proxyLink) close() { | ||
| l.client.Close() | ||
| l.server.Close() | ||
| } | ||
|
|
||
| func (p *blackholeProxy) blackhole() { | ||
| p.mu.Lock() | ||
| defer p.mu.Unlock() | ||
| for _, l := range p.links { | ||
| l.dead.Store(true) | ||
| } | ||
| } | ||
|
|
||
| func (p *blackholeProxy) close() { | ||
| p.ln.Close() | ||
| p.mu.Lock() | ||
| defer p.mu.Unlock() | ||
| for _, l := range p.links { | ||
| l.close() | ||
| } | ||
| } | ||
|
|
||
| // setupDeadConnection returns a GET helper for an HTTP/2 server behind a blackholeProxy, after one | ||
| // successful request whose connection the proxy then blackholes. | ||
| func setupDeadConnection(t *testing.T, transport *http.Transport) (*blackholeProxy, func(time.Duration) (string, error)) { | ||
| srv := httptest.NewUnstartedServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { | ||
| _, _ = io.WriteString(w, r.Proto) | ||
| })) | ||
| srv.EnableHTTP2 = true | ||
| srv.StartTLS() | ||
| t.Cleanup(srv.Close) | ||
|
|
||
| proxy := newBlackholeProxy(t, srv.Listener.Addr().String()) | ||
|
|
||
| transport.TLSClientConfig = srv.Client().Transport.(*http.Transport).TLSClientConfig.Clone() | ||
| t.Cleanup(transport.CloseIdleConnections) | ||
| client := &http.Client{Transport: transport} | ||
| url := "https://" + proxy.ln.Addr().String() + "/" | ||
|
|
||
| get := func(timeout time.Duration) (string, error) { | ||
| ctx, cancel := context.WithTimeout(t.Context(), timeout) | ||
| defer cancel() | ||
| req, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil) | ||
| require.NoError(t, err) | ||
| resp, err := client.Do(req) | ||
| if err != nil { | ||
| return "", err | ||
| } | ||
| defer resp.Body.Close() | ||
| body, err := io.ReadAll(resp.Body) | ||
| return string(body), err | ||
| } | ||
|
|
||
| proto, err := get(5 * time.Second) | ||
| require.NoError(t, err) | ||
| require.Equal(t, "HTTP/2.0", proto) | ||
| require.EqualValues(t, 1, proxy.accepted.Load()) | ||
|
|
||
| proxy.blackhole() | ||
| return proxy, get | ||
| } | ||
|
|
||
| func TestHealthCheckedTransportReplacesDeadConnection(t *testing.T) { | ||
| transport := newHealthCheckedTransport(false) | ||
| transport.HTTP2.SendPingTimeout = time.Second | ||
| transport.HTTP2.PingTimeout = time.Second | ||
| proxy, get := setupDeadConnection(t, transport) | ||
|
|
||
| // A request sent on the dead connection fails once the health check closes it, with an error | ||
| // libs/retry retries. If the health check closed the connection before the request was sent, | ||
| // the request already goes out on a new connection. | ||
| _, err := get(10 * time.Second) | ||
| if err != nil { | ||
| require.ErrorContains(t, err, "http2: client connection lost") | ||
| assert.True(t, retry.IsTransient(t.Context(), err)) | ||
| } | ||
|
|
||
| proto, err := get(10 * time.Second) | ||
| require.NoError(t, err) | ||
| assert.Equal(t, "HTTP/2.0", proto) | ||
| assert.EqualValues(t, 2, proxy.accepted.Load()) | ||
| } | ||
|
|
||
| // Documents the net/http behavior the health check works around: without it, a timed-out request | ||
| // leaves the dead connection in the pool and the next request (a retry) is sent on it again. If a | ||
| // Go release stops reusing such connections, this test fails and can be removed. | ||
| func TestTransportWithoutHealthCheckReusesDeadConnection(t *testing.T) { | ||
| transport := newHealthCheckedTransport(false) | ||
| transport.HTTP2 = nil | ||
| proxy, get := setupDeadConnection(t, transport) | ||
|
|
||
| _, err := get(500 * time.Millisecond) | ||
| require.ErrorIs(t, err, context.DeadlineExceeded) | ||
| _, err = get(500 * time.Millisecond) | ||
| require.ErrorIs(t, err, context.DeadlineExceeded) | ||
| assert.EqualValues(t, 1, proxy.accepted.Load()) | ||
| } | ||
|
|
||
| func TestWorkspaceTransportIsShared(t *testing.T) { | ||
| secure := workspaceTransport(false) | ||
| insecure := workspaceTransport(true) | ||
| assert.Same(t, secure, workspaceTransport(false)) | ||
| assert.Same(t, insecure, workspaceTransport(true)) | ||
| assert.NotSame(t, secure, insecure) | ||
| assert.Nil(t, secure.TLSClientConfig) | ||
| assert.True(t, insecure.TLSClientConfig.InsecureSkipVerify) | ||
| } | ||
|
|
||
| func TestWorkspaceClientUsesHealthCheckedTransport(t *testing.T) { | ||
| setupWorkspaceTest(t) | ||
|
|
||
| err := databrickscfg.SaveToProfile(t.Context(), &config.Config{ | ||
| Profile: "secure", | ||
| Host: "https://secure.cloud.databricks.test", | ||
| Token: "token", | ||
| }) | ||
| require.NoError(t, err) | ||
| // skip_verify comes from the profile, so it is only known after the config is resolved. | ||
| err = databrickscfg.SaveToProfile(t.Context(), &config.Config{ | ||
| Profile: "insecure", | ||
| Host: "https://insecure.cloud.databricks.test", | ||
| Token: "token", | ||
| InsecureSkipVerify: true, | ||
| }) | ||
| require.NoError(t, err) | ||
|
|
||
| w := Workspace{Profile: "secure"} | ||
| client, err := w.Client(t.Context()) | ||
| require.NoError(t, err) | ||
| assert.Same(t, workspaceTransport(false), client.Config.HTTPTransport) | ||
|
|
||
| w = Workspace{Profile: "insecure"} | ||
| client, err = w.Client(t.Context()) | ||
| require.NoError(t, err) | ||
| assert.Same(t, workspaceTransport(true), client.Config.HTTPTransport) | ||
| } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.