Repository navigation
filer: retry small non-seekable uploads to UC Volumes - #7607
Merged
Merged
Conversation
fs cp from a Volume or DBFS to a Volume streams the source download into the Files API upload. #7559 retries uploads only for seekable readers, so a single 503 (e.g. "Azure is temporarily unavailable; try again later.") on that upload still failed the copy. Buffer non-seekable bodies of up to 8 MiB so they can be replayed; larger streams keep one attempt. Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
Collaborator
✅ Integration test reportCommit: ea52c08
Top 3 slowest tests (at least 2 minutes):
|
denik
marked this pull request as ready for review
November 13, 2026 16:51
denik
requested review from
Divyansh-db,
Sankalp-Mittal,
andrewnester and
rclarey
November 13, 2026 16:51
Sankalp-Mittal
left a comment
Contributor
There was a problem hiding this comment.
[AI Generated] Review by review-factory (Claude Opus 5.5 + Claude Fable 5.2, cross-verified). Only points the reviewers agreed on are posted.
A DBFS read handle makes one /dbfs/read call per Read, and io.ReadAll asks for small growing chunks, so buffering a 1 MiB DBFS file took ~20 calls. Read through a 1 MiB bufio.Reader instead. Also update the cmd/fs comment that said Volume writes are not retried. Co-authored-by: Isaac <no-reply@databricks.com>
andrewnester
left a comment
Contributor
There was a problem hiding this comment.
No blocking findings from a full pass; I'll take a manual look at the filer changes before approving.
rclarey
removed their request for review
November 17, 2026 21:41
Co-authored-by: Isaac <no-reply@databricks.com>
andrewnester
left a comment
Contributor
There was a problem hiding this comment.
No blocking findings from a full pass; I'll take a manual look at the filer changes before approving.
andrewnester
approved these changes
Nov 19, 2026
Collaborator
❌ Integration test reportCommit: 44a9c42
1274 interesting tests: 1242 MISS, 9 KNOWN, 9 RECOVERED, 9 FAIL, 5 PANIC
Top 50 slowest tests (at least 2 minutes):
|
idelafuente-oc
pushed a commit
to idelafuente-oc/cli
that referenced
this pull request
Nov 20, 2026
…databricks#7763) ## Changes `DbfsClient.Write` restarts the whole DBFS upload (create, add-block, close) once when add-block or close fails with 404 `RESOURCE_DOES_NOT_EXIST` "No such stream with handle". To restart, it has to read the source again: - seekable sources (local files) seek back to where they started; - non-seekable sources (a download from DBFS or a Volume during `fs cp`, a pipe) are copied into a memory buffer as they are read, up to 8 MiB. Past that limit there is no restart, the same limit databricks#7607 uses for Volumes. The restart always passes overwrite. The first create succeeded, so the file at the path now is this upload's own partial file. Without overwrite, a retried `fs cp` without `--overwrite` would fail on that file with "already exists". Decision: accept the small window where another writer creates the same path between the two attempts and gets overwritten. The testserver's DBFS fake now returns the same message as the real API for an unknown handle. I checked the message on AWS: closing a handle and then calling add-block or close on it returns `404 RESOURCE_DOES_NOT_EXIST "No such stream with handle: [ID]."`. This is separate from databricks#7607, which changes UC Volumes uploads in `files_client.go`. The two buffers look alike, but I kept them apart so the PRs don't depend on each other. ## Why The backend drops stream handles now and then, a few seconds into uploads of files that are only a few bytes. It happens on every cloud, and it failed 38 integration tests (`integration/cmd/fs` and `integration/libs/filer` DBFS variants) in 21 runs on 2026-11-17, including main on Azure. Before this change, `DbfsClient.Write` made a single attempt and the SDK does not retry a 404, so every lost handle failed the command. ## Dashboard | error | snippet | |---|---| | [error/1218](http://go/cli/dashboard/error/1218?main=1&pr=1&end=2026-11-18T12:00:00Z&last=7d) | `dbfs write: No such stream with handle: [ID].` | Example tests: [integration/cmd/fs.TestFsCpFileToDirFileNotOverwritten/uc-volumes_to_dbfs](http://go/cli/dashboard/test/integration/cmd/fs.TestFsCpFileToDirFileNotOverwritten/uc-volumes_to_dbfs?main=1&pr=1&end=2026-11-18T12:00:00Z&last=7d), [integration/cmd/fs.TestFsCpDirToDirWithOverwriteFlag/dbfs_to_dbfs](http://go/cli/dashboard/test/integration/cmd/fs.TestFsCpDirToDirWithOverwriteFlag/dbfs_to_dbfs?main=1&pr=1&end=2026-11-18T12:00:00Z&last=7d) ``` cp_test.go:237: Error: Received unexpected error: dbfs write: No such stream with handle: [ID]. Test: TestFsCpFileToDirFileNotOverwritten/uc-volumes_to_dbfs ``` ## Tests - New unit tests in `libs/filer/dbfs_client_test.go` use a fake that drops the handle on add-block or close. They cover non-seekable and seekable sources, a large source that fails early, the single-restart limit, the buffer limit, and the case where the error is not a lost handle. All but the last fail without the fix. - `TestFsCpFileToDirFileNotOverwritten`, `TestFsCpDirToDirWithOverwriteFlag` and `TestFsCpFileToFileWithOverwriteFlag` pass on AWS (30 subtests). A lost handle could not be triggered against the real backend. This pull request and its description were written by Isaac. --------- Co-authored-by: Isaac <no-reply@databricks.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Changes
FilesClient.Write(UC Volumes) now buffers a non-seekable upload body of up to 8 MiB in memory, so the upload is retried on 408/429/502/503/504 like a seekable one. Bodies over 8 MiB are streamed with one attempt, as before.Why
#7559 retries Files API uploads by rewinding a seekable reader.
fs cpfrom a Volume or from DBFS to a Volume passes the source's download stream toWrite, which is not seekable, so a single 503 on the upload still failed the copy. A local source (*os.File) is seekable and already covered.The errors below are HTTP 503 from the Files API. All failures in the window predate #7556 and #7559 (merged 2026-11-12 ~18:10 UTC). One later PR failure ran on a stacked branch that does not include them. Those PRs cover reads, stats, deletes, mkdir and seekable writes, including the test-setup writes behind most of these failures. This PR closes the remaining gap: remote-to-Volume copies.
Decisions:
cmd/fs, so every caller (fs cp, bundle uploads of generated content) gets it, and the size does not have to be known up front.Open points for the reviewer:
cpFileToFilewould re-open the source withsourceFiler.Readand retry theWrite. That covers files of any size with no buffering, but only forfs cp, andcmd/fswould need its own status-code policy. I went with the smaller, filer-local change. Happy to switch.Tests
TestFilesClientDoesNotRetryNonSeekableUploadwith three tests.TestFilesClientRetriesSmallNonSeekableUploadcovers a body of exactly the buffer size and fails without the fix.TestFilesClientRetriesSmallPipeUploadcovers an*os.Filewhose Seek fails.TestFilesClientDoesNotRetryLargeNonSeekableUploadchecks a larger body gets one attempt and is sent in full. The tests lower the limit to a few bytes.TestFilesClientBuffersUploadInLargeReads: the source is read in 1 MiB chunks, so buffering a DBFS file takes about one/dbfs/readper MiB rather than ~20. It fails without thebufio.Reader.go test -race ./libs/filer/ ./cmd/fs/and localTestAccept/cmd/fspass.fs cpdbfs_to_uc-volumes and uc-volumes_to_uc-volumes variants re-run after the 1 MiB read change: pass.integration/cmd/fsTestFsCpDir,TestFsCpDirToDirFileNotOverwritten,TestFsCpFileToFileFileNotOverwrittenandTestFsRmEmptyDirpass, all variants. A 503 cannot be forced on cloud.Dashboard
databricks-api error: Azure is temporarily unavailable; try again later.(17 failures, 13 tests, Azure)databricks-api error: GCP is temporarily unavailable; try again later.(6 failures, GCP)databricks-api error: The service at /api/2.0/fs/files/[PATH] is temporarily unavailable. Please try again later.Remote-to-Volume tests this PR covers:
AI agent (harness_auto).
This pull request and its description were written by Isaac.