Repository navigation
Retry overwriting file writes on unexpected EOF - #7611
Merged
Merged
Conversation
Writes with OverwriteIfExists are idempotent and already retry 5xx and the SDK inactivity timeout. Also retry the transport errors reads retry (unexpected EOF, REFUSED_STREAM): import-file uploads fail with "unexpected EOF" when the connection is cut mid-request. Co-authored-by: Isaac <no-reply@databricks.com>
Collaborator
✅ Integration test reportCommit: 607ee14
Top 4 slowest tests (at least 2 minutes):
|
andrewnester
approved these changes
Nov 15, 2026
idelafuente-oc
pushed a commit
to idelafuente-oc/cli
that referenced
this pull request
Nov 20, 2026
## Changes
- `Locker.Lock` now retries the lock create (`import-file` with
`overwrite=false`) on the errors `libs/retry.IsTransientRead` covers:
5xx/408 that the SDK did not retry, the SDK's inactivity timeout
("request timed out after ... of inactivity"), and a connection cut
mid-response (`unexpected EOF`, `REFUSED_STREAM`). That check is named
for reads, but the lock create is safe to repeat for the reason below,
so it gets the same coverage. Forced locks overwrite and were already
retried by `retry.Filer` (on `retry.IsTransient`, which does not include
`unexpected EOF`), so their write is unchanged; they do get the
read-back below. Retrying overwrites on `unexpected EOF` is left to
[databricks#7611](databricks#7611).
- When the last attempt still fails with a transient error, forced or
not, `Lock` reads the lock back. If that attempt was applied and the
lock is ours, the lock is taken; otherwise it would stay behind, and
nobody would release it. If the lock belongs to someone else, the error
names the holder ("deploy lock acquired by ..."), except with
`--force-lock`, where that message would suggest the flag already in
use, so the write error is returned instead. If there is no lock, or the
read fails too, the original error is returned.
- `acceptance/bundle/telemetry/deploy-failed-engine-unknown` stubs every
`import-file` with a 500, which now hits the lock-create retries; it
sets `DATABRICKS_BUNDLE_RETRY_INTERVAL_MS` like the other tests that
inject 5xx, so it does not sleep 30s.
- New acceptance test `bundle/deploy/lock-retry` injects 504s on the
lock create and covers these cases:
- the create was applied before the 504: the retry hits 409 and the lock
turns out to be ours;
- the create was not applied: the retry creates the lock;
- every attempt returns a 504, but the first one was applied: the
read-back finds our lock;
- another deployer already holds the lock: the retry hits 409 and the
error names the holder.
- another deployer holds the lock and every attempt returns a 504: the
read-back names the holder;
- `--force-lock` and every overwrite attempt returns a 504 but is
applied: the read-back finds our lock.
- `--force-lock` over another deployer's lock, and every overwrite
attempt returns a 504 without being applied: the error is the 504.
- Unit tests in `libs/locker` for `unexpected EOF` on the lock create:
not applied, applied before the cut, retries exhausted but the last
attempt applied, and every attempt failing. `fault.py` cannot cut a
connection, so this case has no acceptance test.
- New acceptance test `bundle/deploy/lock-timeout`: every lock create
stalls past the inactivity timeout, so the CLI makes three attempts,
finds no lock when it reads it back, and fails with the timeout.
- Changelog entry in `.nextchanges/bundles/lock-retry.md`.
## Why
[databricks#7521](databricks#7521) made workspace
calls retry after the SDK's inactivity timeout, but it deliberately
skipped writes that do not overwrite, because a repeated create could
fail on the file its own earlier attempt created. The deploy lock create
is one of those writes, so the same server-side stalls still fail
deploys at "Failed to acquire deployment lock". The failures below all
ran on a main commit that already includes databricks#7521.
Retrying this create is safe because of how the lock works: the lock
file holds the locker's unique ID, and `Lock` always reads the file back
(`assertLockHeld`) after a create that fails with "already exists". So
whatever the earlier attempt did, the retry ends in a correct state:
- the earlier attempt created the file: the retry gets 409, the
read-back finds our ID, and we hold the lock;
- it did not: the retry creates the file;
- another deployer took the lock in between: the retry gets 409, the
read-back finds their ID, and we fail with the usual "deploy lock
acquired by ..." error.
A late-arriving earlier attempt cannot overwrite anything either, since
it also uses `overwrite=false`.
Known edge case, which the 504 retries elsewhere share: after an
inactivity timeout the server might still apply the stalled request
later. If it does so only after this deploy has released the lock, the
late create leaves a stale lock carrying our ID, and the next deploy
needs `--force-lock`. A stale lock from a single timed-out create was
already possible before this change.
Not done: an opt-in in `retry.Filer` for "fs.ErrExist after a retry is
fine". Only the lock needs it today, and it is safe there only because
of the read-back, so I kept the logic in `Lock`.
Judgment call: I used the same retry budget as databricks#7521 (`MaxRetries = 2`,
15s pause) so the lock acts like the other workspace writes. A
persistent write stall now takes about 5 minutes (3 × 90s + 2 × 15s)
before the deploy fails, which matches databricks#7521. If reads stall too, the
read-back after the last attempt is retried the same way through
`retry.Filer`, so the worst case is about 10 minutes. I kept the retried
read: a single read gives up on the first stall, which more often leaves
an applied lock of ours behind for nobody to release.
## Tests
- `acceptance/bundle/deploy/lock-retry` and `lock-timeout` (local
testserver, both DMS variants). Without the change, it fails with
`Error: Failed to acquire deployment lock: Fault injected by test.`
- Ran the local `bundle/deploy`, `bundle/destroy`, `bundle/upload`,
`bundle/state` and `pipelines` deploy/destroy acceptance tests. The only
failures are `bundle/state/state_present` and
`bundle/state/engine_default`, which fail on main too on my machine:
they record the user agent, which picks up an extra agent-environment
token there.
- I did not add a cloud test: the stalls are random and server-side, so
a cloud run cannot trigger them on demand. The timeout classification is
already unit-tested in `libs/retry`.
## Dashboard:
| error | snippet |
| -- | -- |
|
[error/124](http://go/cli/dashboard/error/124?main=1&pr=1&end=2026-11-13T09:00:00Z&last=2d)
| `Error: Failed to acquire deployment lock: Post
"[DATABRICKS_URL]/api/2.0/workspace-files/import-file/Workspace%2FUsers%2F[USERNAME]%2F.bundle%2F...%2Fdefault%2Fstate%2Fdeploy.lock?overwrite=false":
request timed out after 1m30s of inactivity` |
|
[error/125](http://go/cli/dashboard/error/125?main=1&pr=1&end=2026-11-15T10:00:00Z&last=7d)
| `Error: Failed to acquire deployment lock: Post
"[DATABRICKS_URL]/api/2.0/workspace-files/import-file/[PATH]":
unexpected EOF` |
On main in the last 2 days: 13 failures on 11 tests (AWS, Azure, GCP),
e.g.
[TestAccept/bundle/resources/postgres_branches/recreate/DMS=](http://go/cli/dashboard/test/acceptance.TestAccept/bundle/resources/postgres_branches/recreate/DMS=?main=1&pr=1&end=2026-11-13T09:00:00Z&last=2d)
and
[TestAccept/bundle/resources/grants/feature_specs/DMS=](http://go/cli/dashboard/test/acceptance.TestAccept/bundle/resources/grants/feature_specs/DMS=?main=1&pr=1&end=2026-11-13T09:00:00Z&last=2d).
This pull request and its description were written by Isaac.
---------
Co-authored-by: Isaac <no-reply@databricks.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
File uploads to
POST /api/2.0/workspace-files/import-filesometimes fail with a transport error when the connection is cut mid-request.syncandbundle deploythen fail outright. On Azure this hit 14 tests in the last two weeks, mostly on main:integration/cmd/synctests and bundle acceptance tests that upload files.The retrying filer (
libs/retry) already retries overwriting writes on 5xx and on the SDK inactivity timeout, but withIsTransient, which leaves outunexpected EOF. #7129 addedunexpected EOFandREFUSED_STREAMretries for reads only. An overwrite is just as idempotent, and a 504 or inactivity timeout, which writes already retry, leaves the same uncertainty about whether the server applied the request.Changes
retryingFiler.WritewithOverwriteIfExistsnow usesIsTransientRead, the same check thatRead,ReadDirandStatuse. Writes without overwrite andDeletekeepIsTransient. Doc comments are updated to match.Tests
TestFilerRetriesWriteOnUnexpectedEOF: fails without the change and passes with it.go test ./libs/retry ./libs/sync ./bundle/deploy/...and thecmd/sync,bundle/syncandcmd/fsacceptance tests pass locally.Dashboard
Error: Post "[DATABRICKS_URL]/api/2.0/workspace-files/import-file/[PATH]": unexpected EOFThis pull request and its description were written by Isaac.