Skip to content

Retry overwriting file writes on unexpected EOF - #7611

Merged
denik merged 1 commit into
mainfrom
denik/retry-overwrite-unexpected-eof
Nov 15, 2026
Merged

denik merged 1 commit into
mainfrom
denik/retry-overwrite-unexpected-eof

Conversation

@denik

@denik denik commented Nov 13, 2026

Copy link
Copy Markdown
Contributor

Why

File uploads to POST /api/2.0/workspace-files/import-file sometimes fail with a transport error when the connection is cut mid-request. sync and bundle deploy then fail outright. On Azure this hit 14 tests in the last two weeks, mostly on main: integration/cmd/sync tests and bundle acceptance tests that upload files.

Error: Post "[DATABRICKS_URL]/api/2.0/workspace-files/import-file/[PATH]": unexpected EOF

The retrying filer (libs/retry) already retries overwriting writes on 5xx and on the SDK inactivity timeout, but with IsTransient, which leaves out unexpected EOF. #7129 added unexpected EOF and REFUSED_STREAM retries for reads only. An overwrite is just as idempotent, and a 504 or inactivity timeout, which writes already retry, leaves the same uncertainty about whether the server applied the request.

Changes

retryingFiler.Write with OverwriteIfExists now uses IsTransientRead, the same check that Read, ReadDir and Stat use. Writes without overwrite and Delete keep IsTransient. Doc comments are updated to match.

Tests

  • New unit test TestFilerRetriesWriteOnUnexpectedEOF: fails without the change and passes with it.
  • go test ./libs/retry ./libs/sync ./bundle/deploy/... and the cmd/sync, bundle/sync and cmd/fs acceptance tests pass locally.

Dashboard

error snippet
error/180 Error: Post "[DATABRICKS_URL]/api/2.0/workspace-files/import-file/[PATH]": unexpected EOF

This pull request and its description were written by Isaac.

Writes with OverwriteIfExists are idempotent and already retry 5xx and
the SDK inactivity timeout. Also retry the transport errors reads retry
(unexpected EOF, REFUSED_STREAM): import-file uploads fail with
"unexpected EOF" when the connection is cut mid-request.

Co-authored-by: Isaac <no-reply@databricks.com>
@denik denik added the ai-auto Deprecated: use factory-generated instead. label Nov 13, 2026
@eng-dev-ecosystem-bot

Copy link
Copy Markdown
Collaborator

✅ Integration test report

Commit: 607ee14

Run: 43461429047

Env 💚​RECOVERED ✅​pass 🙈​skip Time
💚​ aws linux-2core-8gb 1 270 15 7:04
💚​ aws-windows-latest-4core-16gb 1 272 13 5:05
💚​ azure linux-2core-8gb 1 269 15 8:10
💚​ azure-windows-latest-4core-16gb 1 271 13 5:38
💚​ gcp linux-2core-8gb 1 270 15 8:26
💚​ gcp-windows-latest-4core-16gb 1 272 13 5:31
Test Name aws linux-2core-8gb aws-windows-latest-4core-16gb azure linux-2core-8gb azure-windows-latest-4core-16gb gcp linux-2core-8gb gcp-windows-latest-4core-16gb
💚​ TestAccept 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R
Top 4 slowest tests (at least 2 minutes):
duration env testname
4:34 aws-windows-latest-4core-16gb TestAccept
3:17 gcp-windows-latest-4core-16gb TestAccept
3:14 azure-windows-latest-4core-16gb TestAccept
2:00 azure linux-2core-8gb TestFilerRecursiveDelete/workspace_files_extensions

@denik
denik marked this pull request as ready for review November 13, 2026 18:18
@denik
denik requested a review from a team as a code owner November 13, 2026 18:18
@denik
denik added this pull request to the merge queue Nov 15, 2026
Merged via the queue into main with commit 20fef7c Nov 15, 2026
28 checks passed
@denik
denik deleted the denik/retry-overwrite-unexpected-eof branch November 15, 2026 19:08
@denik denik added factory-generated and removed ai-auto Deprecated: use factory-generated instead. labels Nov 20, 2026
idelafuente-oc pushed a commit to idelafuente-oc/cli that referenced this pull request Nov 20, 2026
## Changes
- `Locker.Lock` now retries the lock create (`import-file` with
`overwrite=false`) on the errors `libs/retry.IsTransientRead` covers:
5xx/408 that the SDK did not retry, the SDK's inactivity timeout
("request timed out after ... of inactivity"), and a connection cut
mid-response (`unexpected EOF`, `REFUSED_STREAM`). That check is named
for reads, but the lock create is safe to repeat for the reason below,
so it gets the same coverage. Forced locks overwrite and were already
retried by `retry.Filer` (on `retry.IsTransient`, which does not include
`unexpected EOF`), so their write is unchanged; they do get the
read-back below. Retrying overwrites on `unexpected EOF` is left to
[databricks#7611](databricks#7611).
- When the last attempt still fails with a transient error, forced or
not, `Lock` reads the lock back. If that attempt was applied and the
lock is ours, the lock is taken; otherwise it would stay behind, and
nobody would release it. If the lock belongs to someone else, the error
names the holder ("deploy lock acquired by ..."), except with
`--force-lock`, where that message would suggest the flag already in
use, so the write error is returned instead. If there is no lock, or the
read fails too, the original error is returned.
- `acceptance/bundle/telemetry/deploy-failed-engine-unknown` stubs every
`import-file` with a 500, which now hits the lock-create retries; it
sets `DATABRICKS_BUNDLE_RETRY_INTERVAL_MS` like the other tests that
inject 5xx, so it does not sleep 30s.
- New acceptance test `bundle/deploy/lock-retry` injects 504s on the
lock create and covers these cases:
- the create was applied before the 504: the retry hits 409 and the lock
turns out to be ours;
  - the create was not applied: the retry creates the lock;
- every attempt returns a 504, but the first one was applied: the
read-back finds our lock;
- another deployer already holds the lock: the retry hits 409 and the
error names the holder.
- another deployer holds the lock and every attempt returns a 504: the
read-back names the holder;
- `--force-lock` and every overwrite attempt returns a 504 but is
applied: the read-back finds our lock.
- `--force-lock` over another deployer's lock, and every overwrite
attempt returns a 504 without being applied: the error is the 504.
- Unit tests in `libs/locker` for `unexpected EOF` on the lock create:
not applied, applied before the cut, retries exhausted but the last
attempt applied, and every attempt failing. `fault.py` cannot cut a
connection, so this case has no acceptance test.
- New acceptance test `bundle/deploy/lock-timeout`: every lock create
stalls past the inactivity timeout, so the CLI makes three attempts,
finds no lock when it reads it back, and fails with the timeout.
- Changelog entry in `.nextchanges/bundles/lock-retry.md`.

## Why
[databricks#7521](databricks#7521) made workspace
calls retry after the SDK's inactivity timeout, but it deliberately
skipped writes that do not overwrite, because a repeated create could
fail on the file its own earlier attempt created. The deploy lock create
is one of those writes, so the same server-side stalls still fail
deploys at "Failed to acquire deployment lock". The failures below all
ran on a main commit that already includes databricks#7521.

Retrying this create is safe because of how the lock works: the lock
file holds the locker's unique ID, and `Lock` always reads the file back
(`assertLockHeld`) after a create that fails with "already exists". So
whatever the earlier attempt did, the retry ends in a correct state:
- the earlier attempt created the file: the retry gets 409, the
read-back finds our ID, and we hold the lock;
- it did not: the retry creates the file;
- another deployer took the lock in between: the retry gets 409, the
read-back finds their ID, and we fail with the usual "deploy lock
acquired by ..." error.

A late-arriving earlier attempt cannot overwrite anything either, since
it also uses `overwrite=false`.

Known edge case, which the 504 retries elsewhere share: after an
inactivity timeout the server might still apply the stalled request
later. If it does so only after this deploy has released the lock, the
late create leaves a stale lock carrying our ID, and the next deploy
needs `--force-lock`. A stale lock from a single timed-out create was
already possible before this change.

Not done: an opt-in in `retry.Filer` for "fs.ErrExist after a retry is
fine". Only the lock needs it today, and it is safe there only because
of the read-back, so I kept the logic in `Lock`.

Judgment call: I used the same retry budget as databricks#7521 (`MaxRetries = 2`,
15s pause) so the lock acts like the other workspace writes. A
persistent write stall now takes about 5 minutes (3 × 90s + 2 × 15s)
before the deploy fails, which matches databricks#7521. If reads stall too, the
read-back after the last attempt is retried the same way through
`retry.Filer`, so the worst case is about 10 minutes. I kept the retried
read: a single read gives up on the first stall, which more often leaves
an applied lock of ours behind for nobody to release.

## Tests
- `acceptance/bundle/deploy/lock-retry` and `lock-timeout` (local
testserver, both DMS variants). Without the change, it fails with
`Error: Failed to acquire deployment lock: Fault injected by test.`
- Ran the local `bundle/deploy`, `bundle/destroy`, `bundle/upload`,
`bundle/state` and `pipelines` deploy/destroy acceptance tests. The only
failures are `bundle/state/state_present` and
`bundle/state/engine_default`, which fail on main too on my machine:
they record the user agent, which picks up an extra agent-environment
token there.
- I did not add a cloud test: the stalls are random and server-side, so
a cloud run cannot trigger them on demand. The timeout classification is
already unit-tested in `libs/retry`.

## Dashboard:
| error | snippet |
| -- | -- |
|
[error/124](http://go/cli/dashboard/error/124?main=1&pr=1&end=2026-11-13T09:00:00Z&last=2d)
| `Error: Failed to acquire deployment lock: Post
"[DATABRICKS_URL]/api/2.0/workspace-files/import-file/Workspace%2FUsers%2F[USERNAME]%2F.bundle%2F...%2Fdefault%2Fstate%2Fdeploy.lock?overwrite=false":
request timed out after 1m30s of inactivity` |
|
[error/125](http://go/cli/dashboard/error/125?main=1&pr=1&end=2026-11-15T10:00:00Z&last=7d)
| `Error: Failed to acquire deployment lock: Post
"[DATABRICKS_URL]/api/2.0/workspace-files/import-file/[PATH]":
unexpected EOF` |

On main in the last 2 days: 13 failures on 11 tests (AWS, Azure, GCP),
e.g.
[TestAccept/bundle/resources/postgres_branches/recreate/DMS=](http://go/cli/dashboard/test/acceptance.TestAccept/bundle/resources/postgres_branches/recreate/DMS=?main=1&pr=1&end=2026-11-13T09:00:00Z&last=2d)
and
[TestAccept/bundle/resources/grants/feature_specs/DMS=](http://go/cli/dashboard/test/acceptance.TestAccept/bundle/resources/grants/feature_specs/DMS=?main=1&pr=1&end=2026-11-13T09:00:00Z&last=2d).

This pull request and its description were written by Isaac.

---------

Co-authored-by: Isaac <no-reply@databricks.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants