Repository navigation
filer: restart a DBFS upload when the backend loses its stream handle - #7763
Merged
Merged
Conversation
DBFS add-block and close sometimes fail with 404 RESOURCE_DOES_NOT_EXIST "No such stream with handle: N." a few seconds into an upload of a small file. Restart the whole create/add-block/close sequence once, replaying the source: seekable sources seek back, non-seekable ones are replayed from a buffer of up to 8 MiB. Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
Collaborator
✅ Integration test reportCommit: fab4ad8
Top 7 slowest tests (at least 2 minutes):
|
denik
marked this pull request as ready for review
November 18, 2026 12:20
denik
requested review from
Sankalp-Mittal,
andrewnester,
maksim-zuev-db and
rugpanov
and removed request for
maksim-zuev-db
November 18, 2026 12:20
andrewnester
left a comment
Contributor
There was a problem hiding this comment.
No blocking findings from a full pass; I'll take a manual look at the filer changes before approving.
andrewnester
approved these changes
Nov 19, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Changes
DbfsClient.Writerestarts the whole DBFS upload (create, add-block, close) once when add-block or close fails with 404RESOURCE_DOES_NOT_EXIST"No such stream with handle". To restart, it has to read the source again:fs cp, a pipe) are copied into a memory buffer as they are read, up to 8 MiB. Past that limit there is no restart, the same limit filer: retry small non-seekable uploads to UC Volumes #7607 uses for Volumes.The restart always passes overwrite. The first create succeeded, so the file at the path now is this upload's own partial file. Without overwrite, a retried
fs cpwithout--overwritewould fail on that file with "already exists". Decision: accept the small window where another writer creates the same path between the two attempts and gets overwritten.The testserver's DBFS fake now returns the same message as the real API for an unknown handle. I checked the message on AWS: closing a handle and then calling add-block or close on it returns
404 RESOURCE_DOES_NOT_EXIST "No such stream with handle: [ID].".This is separate from #7607, which changes UC Volumes uploads in
files_client.go. The two buffers look alike, but I kept them apart so the PRs don't depend on each other.Why
The backend drops stream handles now and then, a few seconds into uploads of files that are only a few bytes. It happens on every cloud, and it failed 38 integration tests (
integration/cmd/fsandintegration/libs/filerDBFS variants) in 21 runs on 2026-11-17, including main on Azure. Before this change,DbfsClient.Writemade a single attempt and the SDK does not retry a 404, so every lost handle failed the command.Dashboard
dbfs write: No such stream with handle: [ID].Example tests: integration/cmd/fs.TestFsCpFileToDirFileNotOverwritten/uc-volumes_to_dbfs, integration/cmd/fs.TestFsCpDirToDirWithOverwriteFlag/dbfs_to_dbfs
Tests
libs/filer/dbfs_client_test.gouse a fake that drops the handle on add-block or close. They cover non-seekable and seekable sources, a large source that fails early, the single-restart limit, the buffer limit, and the case where the error is not a lost handle. All but the last fail without the fix.TestFsCpFileToDirFileNotOverwritten,TestFsCpDirToDirWithOverwriteFlagandTestFsCpFileToFileWithOverwriteFlagpass on AWS (30 subtests). A lost handle could not be triggered against the real backend.This pull request and its description were written by Isaac.