Skip to content

filer: restart a DBFS upload when the backend loses its stream handle - #7763

Merged
denik merged 2 commits into
mainfrom
dbfs-retry-lost-handle
Nov 19, 2026
Merged

denik merged 2 commits into
mainfrom
dbfs-retry-lost-handle

Conversation

@denik

@denik denik commented Nov 18, 2026

Copy link
Copy Markdown
Contributor

Changes

DbfsClient.Write restarts the whole DBFS upload (create, add-block, close) once when add-block or close fails with 404 RESOURCE_DOES_NOT_EXIST "No such stream with handle". To restart, it has to read the source again:

  • seekable sources (local files) seek back to where they started;
  • non-seekable sources (a download from DBFS or a Volume during fs cp, a pipe) are copied into a memory buffer as they are read, up to 8 MiB. Past that limit there is no restart, the same limit filer: retry small non-seekable uploads to UC Volumes #7607 uses for Volumes.

The restart always passes overwrite. The first create succeeded, so the file at the path now is this upload's own partial file. Without overwrite, a retried fs cp without --overwrite would fail on that file with "already exists". Decision: accept the small window where another writer creates the same path between the two attempts and gets overwritten.

The testserver's DBFS fake now returns the same message as the real API for an unknown handle. I checked the message on AWS: closing a handle and then calling add-block or close on it returns 404 RESOURCE_DOES_NOT_EXIST "No such stream with handle: [ID].".

This is separate from #7607, which changes UC Volumes uploads in files_client.go. The two buffers look alike, but I kept them apart so the PRs don't depend on each other.

Why

The backend drops stream handles now and then, a few seconds into uploads of files that are only a few bytes. It happens on every cloud, and it failed 38 integration tests (integration/cmd/fs and integration/libs/filer DBFS variants) in 21 runs on 2026-11-17, including main on Azure. Before this change, DbfsClient.Write made a single attempt and the SDK does not retry a 404, so every lost handle failed the command.

Dashboard

error snippet
error/1218 dbfs write: No such stream with handle: [ID].

Example tests: integration/cmd/fs.TestFsCpFileToDirFileNotOverwritten/uc-volumes_to_dbfs, integration/cmd/fs.TestFsCpDirToDirWithOverwriteFlag/dbfs_to_dbfs

cp_test.go:237:
    Error:      Received unexpected error:
                dbfs write: No such stream with handle: [ID].
    Test:       TestFsCpFileToDirFileNotOverwritten/uc-volumes_to_dbfs

Tests

  • New unit tests in libs/filer/dbfs_client_test.go use a fake that drops the handle on add-block or close. They cover non-seekable and seekable sources, a large source that fails early, the single-restart limit, the buffer limit, and the case where the error is not a lost handle. All but the last fail without the fix.
  • TestFsCpFileToDirFileNotOverwritten, TestFsCpDirToDirWithOverwriteFlag and TestFsCpFileToFileWithOverwriteFlag pass on AWS (30 subtests). A lost handle could not be triggered against the real backend.

This pull request and its description were written by Isaac.

DBFS add-block and close sometimes fail with 404 RESOURCE_DOES_NOT_EXIST
"No such stream with handle: N." a few seconds into an upload of a small
file. Restart the whole create/add-block/close sequence once, replaying the
source: seekable sources seek back, non-seekable ones are replayed from a
buffer of up to 8 MiB.

Co-authored-by: Isaac <no-reply@databricks.com>
@denik denik added the ai-auto Deprecated: use factory-generated instead. label Nov 18, 2026
Co-authored-by: Isaac <no-reply@databricks.com>
@eng-dev-ecosystem-bot

Copy link
Copy Markdown
Collaborator

✅ Integration test report

Commit: fab4ad8

Run: 44298057682

Env 💚​RECOVERED ✅​pass 🙈​skip Time
💚​ aws linux-2core-8gb 1 272 19 6:38
💚​ aws-windows-latest-4core-16gb 1 274 17 8:04
💚​ azure linux-2core-8gb 1 271 19 6:54
💚​ azure-windows-latest-4core-16gb 1 273 17 7:00
💚​ gcp linux-2core-8gb 1 272 19 7:26
💚​ gcp-windows-latest-4core-16gb 1 274 17 6:52
Test Name aws linux-2core-8gb aws-windows-latest-4core-16gb azure linux-2core-8gb azure-windows-latest-4core-16gb gcp linux-2core-8gb gcp-windows-latest-4core-16gb
💚​ TestAccept 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R
Top 7 slowest tests (at least 2 minutes):
duration env testname
3:53 aws-windows-latest-4core-16gb TestAccept
3:36 azure-windows-latest-4core-16gb TestAccept
3:32 gcp-windows-latest-4core-16gb TestAccept
2:22 aws-windows-latest-4core-16gb TestFilerWorkspaceFilesExtensionsReadDir
2:12 gcp-windows-latest-4core-16gb TestFilerWorkspaceFilesExtensionsReadDir
2:02 azure-windows-latest-4core-16gb TestFilerWorkspaceFilesExtensionsStat
2:02 gcp linux-2core-8gb TestExportDir

@denik
denik marked this pull request as ready for review November 18, 2026 12:20
@denik
denik requested review from a team as code owners November 18, 2026 12:20
@denik
denik requested review from Sankalp-Mittal, andrewnester, maksim-zuev-db and rugpanov and removed request for maksim-zuev-db November 18, 2026 12:20

@review-oss-bot review-oss-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me — no issues found.

@andrewnester andrewnester left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No blocking findings from a full pass; I'll take a manual look at the filer changes before approving.

@denik
denik added this pull request to the merge queue Nov 19, 2026
Merged via the queue into main with commit 1b83032 Nov 19, 2026
30 checks passed
@denik
denik deleted the dbfs-retry-lost-handle branch November 19, 2026 16:11
@denik denik added factory-generated and removed ai-auto Deprecated: use factory-generated instead. labels Nov 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants