Skip to content

filer: retry small non-seekable uploads to UC Volumes - #7607

Merged
denik merged 5 commits into
mainfrom
denik/filer-retry-small-streams
Nov 19, 2026
Merged

denik merged 5 commits into
mainfrom
denik/filer-retry-small-streams

Conversation

@denik

@denik denik commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

Changes

FilesClient.Write (UC Volumes) now buffers a non-seekable upload body of up to 8 MiB in memory, so the upload is retried on 408/429/502/503/504 like a seekable one. Bodies over 8 MiB are streamed with one attempt, as before.

Why

#7559 retries Files API uploads by rewinding a seekable reader. fs cp from a Volume or from DBFS to a Volume passes the source's download stream to Write, which is not seekable, so a single 503 on the upload still failed the copy. A local source (*os.File) is seekable and already covered.

The errors below are HTTP 503 from the Files API. All failures in the window predate #7556 and #7559 (merged 2026-11-12 ~18:10 UTC). One later PR failure ran on a stacked branch that does not include them. Those PRs cover reads, stats, deletes, mkdir and seekable writes, including the test-setup writes behind most of these failures. This PR closes the remaining gap: remote-to-Volume copies.

Decisions:

  • Buffer in the filer, not in cmd/fs, so every caller (fs cp, bundle uploads of generated content) gets it, and the size does not have to be known up front.
  • 8 MiB keeps memory bounded per upload (fs cp runs a bounded number of copies at a time). #7556 chose not to buffer multi-GB streams, and this keeps that.
  • A retried upload without overwrite can return "already exists" if the first attempt was applied. filer: retry throttled Files API calls on UC Volumes #7559 accepted that for seekable readers, and the same applies here.

Open points for the reviewer:

  • Another option is to retry at the copy layer: cpFileToFile would re-open the source with sourceFiler.Read and retry the Write. That covers files of any size with no buffering, but only for fs cp, and cmd/fs would need its own status-code policy. I went with the smaller, filer-local change. Happy to switch.
  • Files of 8 MiB or less are now fully downloaded before their upload starts, instead of streaming download into upload. For small files this costs little.

Tests

  • Replaced TestFilesClientDoesNotRetryNonSeekableUpload with three tests. TestFilesClientRetriesSmallNonSeekableUpload covers a body of exactly the buffer size and fails without the fix. TestFilesClientRetriesSmallPipeUpload covers an *os.File whose Seek fails. TestFilesClientDoesNotRetryLargeNonSeekableUpload checks a larger body gets one attempt and is sent in full. The tests lower the limit to a few bytes.
  • TestFilesClientBuffersUploadInLargeReads: the source is read in 1 MiB chunks, so buffering a DBFS file takes about one /dbfs/read per MiB rather than ~20. It fails without the bufio.Reader.
  • go test -race ./libs/filer/ ./cmd/fs/ and local TestAccept/cmd/fs pass.
  • On AWS, fs cp dbfs_to_uc-volumes and uc-volumes_to_uc-volumes variants re-run after the 1 MiB read change: pass.
  • On AWS, integration/cmd/fs TestFsCpDir, TestFsCpDirToDirFileNotOverwritten, TestFsCpFileToFileFileNotOverwritten and TestFsRmEmptyDir pass, all variants. A 503 cannot be forced on cloud.

Dashboard

error verbatim
error/1140 databricks-api error: Azure is temporarily unavailable; try again later. (17 failures, 13 tests, Azure)
error/1142 databricks-api error: GCP is temporarily unavailable; try again later. (6 failures, GCP)
error/3319 databricks-api error: The service at /api/2.0/fs/files/[PATH] is temporarily unavailable. Please try again later.

Remote-to-Volume tests this PR covers:

AI agent (harness_auto).

This pull request and its description were written by Isaac.

fs cp from a Volume or DBFS to a Volume streams the source download into
the Files API upload. #7559 retries uploads only for seekable readers, so a
single 503 (e.g. "Azure is temporarily unavailable; try again later.") on
that upload still failed the copy. Buffer non-seekable bodies of up to
8 MiB so they can be replayed; larger streams keep one attempt.

Co-authored-by: Isaac <no-reply@databricks.com>
@denik denik added the ai-auto Deprecated: use factory-generated instead. label Nov 13, 2026
denik and others added 2 commits November 13, 2026 16:14
Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
@eng-dev-ecosystem-bot

eng-dev-ecosystem-bot commented Nov 13, 2026 •

Copy link
Copy Markdown
Collaborator

✅ Integration test report

Commit: ea52c08

Run: 44293917331

Env 💚​RECOVERED ✅​pass 🙈​skip Time
💚​ aws linux-2core-8gb 1 272 19 6:09
💚​ aws-windows-latest-4core-16gb 1 274 17 5:26
💚​ azure linux-2core-8gb 1 271 19 5:51
💚​ azure-windows-latest-4core-16gb 1 273 17 6:49
💚​ gcp linux-2core-8gb 1 272 19 5:53
💚​ gcp-windows-latest-4core-16gb 1 274 17 5:19
Test Name aws linux-2core-8gb aws-windows-latest-4core-16gb azure linux-2core-8gb azure-windows-latest-4core-16gb gcp linux-2core-8gb gcp-windows-latest-4core-16gb
💚​ TestAccept 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R
Top 3 slowest tests (at least 2 minutes):
duration env testname
3:58 azure-windows-latest-4core-16gb TestAccept
3:48 aws-windows-latest-4core-16gb TestAccept
3:34 gcp-windows-latest-4core-16gb TestAccept

@denik
denik marked this pull request as ready for review November 13, 2026 16:51
@denik
denik requested review from a team as code owners November 13, 2026 16:51

@Sankalp-Mittal Sankalp-Mittal left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[AI Generated] Review by review-factory (Claude Opus 5.5 + Claude Fable 5.2, cross-verified). Only points the reviewers agreed on are posted.

Comment thread libs/filer/files_client.go Resolved
Comment thread libs/filer/files_client.go Resolved
A DBFS read handle makes one /dbfs/read call per Read, and io.ReadAll
asks for small growing chunks, so buffering a 1 MiB DBFS file took ~20
calls. Read through a 1 MiB bufio.Reader instead. Also update the
cmd/fs comment that said Volume writes are not retried.

Co-authored-by: Isaac <no-reply@databricks.com>
@denik
denik requested a review from Sankalp-Mittal November 17, 2026 11:59

@andrewnester andrewnester left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No blocking findings from a full pass; I'll take a manual look at the filer changes before approving.

@rclarey
rclarey removed their request for review November 17, 2026 21:41
Comment thread libs/filer/files_client.go Resolved
Comment thread libs/filer/files_client.go Resolved
Co-authored-by: Isaac <no-reply@databricks.com>

@andrewnester andrewnester left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No blocking findings from a full pass; I'll take a manual look at the filer changes before approving.

@denik
denik added this pull request to the merge queue Nov 19, 2026
Merged via the queue into main with commit 44a9c42 Nov 19, 2026
28 checks passed
@denik
denik deleted the denik/filer-retry-small-streams branch November 19, 2026 16:12
@eng-dev-ecosystem-bot

Copy link
Copy Markdown
Collaborator

❌ Integration test report

Commit: 44a9c42

Run: 44563571657

Env 💥​PANIC ❌​FAIL 🟨​KNOWN 🤯​MISS 💚​RECOVERED ✅​pass 🙈​skip Time
💥​ aws linux-2core-8gb 1 630 15 984 1411 320:00
💥​ aws-windows-latest-4core-16gb 1 2 513 21 1009 1439 320:00
💥​ azure linux-2core-8gb 1 1 2 663 13 930 1419 320:00
💥​ azure-windows-latest-4core-16gb 1 8 584 12 921 1447 320:00
💥​ gcp linux-2core-8gb 1 4 2 505 18 1087 1417 320:00
❌​ gcp-windows-latest-4core-16gb 2 600 21 910 1445 320:00
1274 interesting tests: 1242 MISS, 9 KNOWN, 9 RECOVERED, 9 FAIL, 5 PANIC
Test Name aws linux-2core-8gb aws-windows-latest-4core-16gb azure linux-2core-8gb azure-windows-latest-4core-16gb gcp linux-2core-8gb gcp-windows-latest-4core-16gb
🤯​ TestAccept 🤯​M 🤯​M 🤯​M 🤯​M 🤯​M 🤯​M
🤯​ TestAccept/bundle/apps/job_permissions 🤯​M 🤯​M 🤯​M 🤯​M 🤯​M ✅​p
🤯​ TestAccept/bundle/apps/job_permissions/DMS= 🤯​M 🤯​M 🤯​M 🤯​M 🤯​M ✅​p
🤯​ TestAccept/bundle/apps/job_permissions/DMS=true 🤯​M 🤯​M 🤯​M 🤯​M 🤯​M ✅​p
🤯​ TestAccept/bundle/artifacts/artifact_path_with_volume/volume_doesnot_exist 🤯​M 🤯​M 🤯​M 🤯​M 🤯​M ✅​p
🤯​ TestAccept/bundle/artifacts/artifact_path_with_volume/volume_doesnot_exist/DMS= 🤯​M 🤯​M 🤯​M 🤯​M 🤯​M ✅​p
🤯​ TestAccept/bundle/artifacts/artifact_path_with_volume/volume_doesnot_exist/DMS=true 🤯​M 🤯​M 🤯​M 🤯​M 🤯​M ✅​p
🤯​ TestAccept/bundle/artifacts/artifact_path_with_volume/volume_not_deployed 🤯​M ✅​p 🤯​M 🤯​M 🤯​M ✅​p
🤯​ TestAccept/bundle/artifacts/artifact_path_with_volume/volume_not_deployed/DMS= 🤯​M ✅​p 🤯​M 🤯​M 🤯​M ✅​p
🤯​ TestAccept/bundle/artifacts/artifact_path_with_volume/volume_not_deployed/DMS=true 🤯​M ✅​p 🤯​M 🤯​M 🤯​M ✅​p
🤯​ TestAccept/bundle/config-remote-sync/cli_defaults 🤯​M 🙈​s 🤯​M 🙈​s 🤯​M 🙈​s
🤯​ TestAccept/bundle/config-remote-sync/cli_defaults/DMS= 🤯​M 🤯​M 🤯​M
🤯​ TestAccept/bundle/config-remote-sync/cli_defaults/DMS=true 🤯​M 🤯​M 🤯​M
🤯​ TestAccept/bundle/config-remote-sync/cluster_policy_cross_target 🤯​M 🙈​s 🤯​M 🙈​s 🤯​M 🙈​s
🤯​ TestAccept/bundle/config-remote-sync/cluster_policy_cross_target/DMS= 🤯​M 🤯​M 🤯​M
🤯​ TestAccept/bundle/config-remote-sync/cluster_policy_cross_target/DMS=true 🤯​M 🤯​M 🤯​M
🤯​ TestAccept/bundle/config-remote-sync/cluster_policy_permissions 🤯​M 🙈​s 🤯​M 🙈​s 🤯​M 🙈​s
🤯​ TestAccept/bundle/config-remote-sync/cluster_policy_permissions/DMS= 🤯​M 🤯​M 🤯​M
🤯​ TestAccept/bundle/config-remote-sync/cluster_policy_permissions/DMS=true 🤯​M 🤯​M 🤯​M
🤯​ TestAccept/bundle/config-remote-sync/cluster_policy_remote_addition 🤯​M 🙈​s 🤯​M 🙈​s 🤯​M 🙈​s
🤯​ TestAccept/bundle/config-remote-sync/cluster_policy_remote_addition/DMS= 🤯​M 🤯​M 🤯​M
🤯​ TestAccept/bundle/config-remote-sync/cluster_policy_remote_addition/DMS=true 🤯​M 🤯​M 🤯​M
🤯​ TestAccept/bundle/config-remote-sync/config_edits 🤯​M 🙈​s 🤯​M 🙈​s 🤯​M 🙈​s
🤯​ TestAccept/bundle/config-remote-sync/config_edits/DMS= 🤯​M 🤯​M 🤯​M
🤯​ TestAccept/bundle/config-remote-sync/config_edits/DMS=true 🤯​M 🤯​M 🤯​M
🤯​ TestAccept/bundle/config-remote-sync/dashboard_etag 🤯​M 🙈​s 🤯​M 🙈​s 🤯​M 🙈​s
🤯​ TestAccept/bundle/config-remote-sync/dashboard_etag/DMS= 🤯​M 🤯​M 🤯​M
🤯​ TestAccept/bundle/config-remote-sync/dashboard_etag/DMS=true 🤯​M 🤯​M 🤯​M
🤯​ TestAccept/bundle/config-remote-sync/external_cluster_policy_cross_target 🤯​M 🙈​s 🤯​M 🙈​s 🤯​M 🙈​s
🤯​ TestAccept/bundle/config-remote-sync/external_cluster_policy_cross_target/DMS= 🤯​M 🤯​M 🤯​M
🤯​ TestAccept/bundle/config-remote-sync/external_cluster_policy_cross_target/DMS=true 🤯​M 🤯​M 🤯​M
🤯​ TestAccept/bundle/config-remote-sync/flushed_cache 🤯​M 🙈​s 🤯​M 🙈​s 🤯​M 🙈​s
🤯​ TestAccept/bundle/config-remote-sync/flushed_cache/DMS= 🤯​M 🤯​M 🤯​M
🤯​ TestAccept/bundle/config-remote-sync/flushed_cache/DMS=true 🤯​M 🤯​M 🤯​M
🤯​ TestAccept/bundle/config-remote-sync/formatting_preserved 🤯​M 🙈​s 🤯​M 🙈​s 🤯​M 🙈​s
🤯​ TestAccept/bundle/config-remote-sync/formatting_preserved/DMS= 🤯​M 🤯​M 🤯​M
🤯​ TestAccept/bundle/config-remote-sync/formatting_preserved/DMS=true 🤯​M 🤯​M 🤯​M
🤯​ TestAccept/bundle/config-remote-sync/job_fields 🤯​M 🙈​s 🤯​M 🙈​s 🤯​M 🙈​s
🤯​ TestAccept/bundle/config-remote-sync/job_fields/DMS= 🤯​M 🤯​M 🤯​M
🤯​ TestAccept/bundle/config-remote-sync/job_fields/DMS=true 🤯​M 🤯​M 🤯​M
🤯​ TestAccept/bundle/config-remote-sync/job_multiple_tasks 🤯​M 🙈​s 🤯​M 🙈​s 🤯​M 🙈​s
🤯​ TestAccept/bundle/config-remote-sync/job_multiple_tasks/DMS= 🤯​M 🤯​M 🤯​M
🤯​ TestAccept/bundle/config-remote-sync/job_multiple_tasks/DMS=true 🤯​M 🤯​M 🤯​M
🤯​ TestAccept/bundle/config-remote-sync/job_params_variables 🤯​M 🙈​s 🤯​M 🙈​s 🤯​M 🙈​s
🤯​ TestAccept/bundle/config-remote-sync/job_params_variables/DMS= 🤯​M 🤯​M 🤯​M
🤯​ TestAccept/bundle/config-remote-sync/job_params_variables/DMS=true 🤯​M 🤯​M 🤯​M
🤯​ TestAccept/bundle/config-remote-sync/job_pipeline_task 🤯​M 🙈​s 🤯​M 🙈​s 🤯​M 🙈​s
🤯​ TestAccept/bundle/config-remote-sync/job_pipeline_task/DMS= 🤯​M 🤯​M 🤯​M
🤯​ TestAccept/bundle/config-remote-sync/job_pipeline_task/DMS=true 🤯​M 🤯​M 🤯​M
🤯​ TestAccept/bundle/config-remote-sync/json_status_missing_state 🤯​M 🙈​s 🤯​M 🙈​s 🤯​M 🙈​s
🤯​ TestAccept/bundle/config-remote-sync/json_status_missing_state/DMS= 🤯​M 🤯​M 🤯​M
🤯​ TestAccept/bundle/config-remote-sync/json_status_missing_state/DMS=true 🤯​M 🤯​M 🤯​M
🤯​ TestAccept/bundle/config-remote-sync/json_status_selectors 🤯​M 🙈​s 🤯​M 🙈​s 🤯​M 🙈​s
🤯​ TestAccept/bundle/config-remote-sync/json_status_selectors/DMS= 🤯​M 🤯​M 🤯​M
🤯​ TestAccept/bundle/config-remote-sync/json_status_selectors/DMS=true 🤯​M 🤯​M 🤯​M
🤯​ TestAccept/bundle/config-remote-sync/multiple_files 🤯​M 🙈​s 🤯​M 🙈​s 🤯​M 🙈​s
🤯​ TestAccept/bundle/config-remote-sync/multiple_files/DMS= 🤯​M 🤯​M 🤯​M
🤯​ TestAccept/bundle/config-remote-sync/multiple_files/DMS=true 🤯​M 🤯​M 🤯​M
🤯​ TestAccept/bundle/config-remote-sync/multiple_resources 🤯​M 🙈​s 🤯​M 🙈​s 🤯​M 🙈​s
🤯​ TestAccept/bundle/config-remote-sync/multiple_resources/DMS= 🤯​M 🤯​M 🤯​M
🤯​ TestAccept/bundle/config-remote-sync/multiple_resources/DMS=true 🤯​M 🤯​M 🤯​M
🤯​ TestAccept/bundle/config-remote-sync/output_json 🤯​M 🙈​s 🤯​M 🙈​s 🤯​M 🙈​s
🤯​ TestAccept/bundle/config-remote-sync/output_json/DMS= 🤯​M 🤯​M 🤯​M
🤯​ TestAccept/bundle/config-remote-sync/output_json/DMS=true 🤯​M 🤯​M 🤯​M
🤯​ TestAccept/bundle/config-remote-sync/output_no_changes 🤯​M 🙈​s 🤯​M 🙈​s 🤯​M 🙈​s
🤯​ TestAccept/bundle/config-remote-sync/output_no_changes/DMS= 🤯​M 🤯​M 🤯​M
🤯​ TestAccept/bundle/config-remote-sync/output_no_changes/DMS=true 🤯​M 🤯​M 🤯​M
🤯​ TestAccept/bundle/config-remote-sync/pipeline_fields 🤯​M 🙈​s 🤯​M 🙈​s 🤯​M 🙈​s
🤯​ TestAccept/bundle/config-remote-sync/pipeline_fields/DMS= 🤯​M 🤯​M 🤯​M
🤯​ TestAccept/bundle/config-remote-sync/pipeline_fields/DMS=true 🤯​M 🤯​M 🤯​M
🤯​ TestAccept/bundle/config-remote-sync/policy_injected_cluster_fields 🤯​M 🙈​s 🤯​M 🙈​s 🤯​M 🙈​s
🤯​ TestAccept/bundle/config-remote-sync/policy_injected_cluster_fields/DMS= 🤯​M 🤯​M 🤯​M
🤯​ TestAccept/bundle/config-remote-sync/policy_injected_cluster_fields/DMS=true 🤯​M 🤯​M 🤯​M
🤯​ TestAccept/bundle/config-remote-sync/resolve_variables 🤯​M 🙈​s 🤯​M 🙈​s 🤯​M 🙈​s
🤯​ TestAccept/bundle/config-remote-sync/resolve_variables/DMS= 🤯​M 🤯​M 🤯​M
🤯​ TestAccept/bundle/config-remote-sync/resolve_variables/DMS=true 🤯​M 🤯​M 🤯​M
🤯​ TestAccept/bundle/config-remote-sync/select_basic 🤯​M 🙈​s 🤯​M 🙈​s 🤯​M 🙈​s
🤯​ TestAccept/bundle/config-remote-sync/select_basic/DMS= 🤯​M 🤯​M 🤯​M
🤯​ TestAccept/bundle/config-remote-sync/select_basic/DMS=true 🤯​M 🤯​M 🤯​M
🤯​ TestAccept/bundle/config-remote-sync/select_multiple 🤯​M 🙈​s 🤯​M 🙈​s 🤯​M 🙈​s
🤯​ TestAccept/bundle/config-remote-sync/select_multiple/DMS= 🤯​M 🤯​M 🤯​M
🤯​ TestAccept/bundle/config-remote-sync/select_multiple/DMS=true 🤯​M 🤯​M 🤯​M
🤯​ TestAccept/bundle/config-remote-sync/skip_deployment 🤯​M 🙈​s 🤯​M 🙈​s 🤯​M 🙈​s
🤯​ TestAccept/bundle/config-remote-sync/skip_deployment/DMS= 🤯​M 🤯​M 🤯​M
🤯​ TestAccept/bundle/config-remote-sync/skip_deployment/DMS=true 🤯​M 🤯​M 🤯​M
🤯​ TestAccept/bundle/config-remote-sync/skip_permissions 🤯​M 🙈​s 🤯​M 🙈​s 🤯​M 🙈​s
🤯​ TestAccept/bundle/config-remote-sync/skip_permissions/DMS= 🤯​M 🤯​M 🤯​M
🤯​ TestAccept/bundle/config-remote-sync/skip_permissions/DMS=true 🤯​M 🤯​M 🤯​M
🤯​ TestAccept/bundle/config-remote-sync/target_override 🤯​M 🙈​s 🤯​M 🙈​s 🤯​M 🙈​s
🤯​ TestAccept/bundle/config-remote-sync/target_override/DMS= 🤯​M 🤯​M 🤯​M
🤯​ TestAccept/bundle/config-remote-sync/target_override/DMS=true 🤯​M 🤯​M 🤯​M
🤯​ TestAccept/bundle/config-remote-sync/task_rename_revert 🤯​M 🙈​s 🤯​M 🙈​s 🤯​M 🙈​s
🤯​ TestAccept/bundle/config-remote-sync/task_rename_revert/DMS= 🤯​M 🤯​M 🤯​M
🤯​ TestAccept/bundle/config-remote-sync/task_rename_revert/DMS=true 🤯​M 🤯​M 🤯​M
🤯​ TestAccept/bundle/debug/fetch-repository-info 🤯​M ✅​p 🤯​M 🤯​M 🤯​M ✅​p
🤯​ TestAccept/bundle/debug/fetch-repository-info/DMS= 🤯​M ✅​p 🤯​M 🤯​M 🤯​M ✅​p
🤯​ TestAccept/bundle/debug/fetch-repository-info/DMS=true 🤯​M ✅​p 🤯​M 🤯​M 🤯​M ✅​p
🤯​ TestAccept/bundle/deploy/empty-bundle 🤯​M ✅​p 🤯​M 🤯​M 🤯​M ✅​p
🤯​ TestAccept/bundle/deploy/empty-bundle/DATABRICKS_BUNDLE_ENABLE_EXPERIMENTAL_YAML_SYNC=/DMS= 🤯​M ✅​p 🤯​M 🤯​M 🤯​M ✅​p
🤯​ TestAccept/bundle/deploy/empty-bundle/DATABRICKS_BUNDLE_ENABLE_EXPERIMENTAL_YAML_SYNC=/DMS=true 🤯​M ✅​p 🤯​M 🤯​M 🤯​M ✅​p
🤯​ TestAccept/bundle/deploy/empty-bundle/DATABRICKS_BUNDLE_ENABLE_EXPERIMENTAL_YAML_SYNC=true/DMS= 🤯​M ✅​p 🤯​M 🤯​M 🤯​M ✅​p
🤯​ TestAccept/bundle/deploy/empty-bundle/DATABRICKS_BUNDLE_ENABLE_EXPERIMENTAL_YAML_SYNC=true/DMS=true 🤯​M ✅​p 🤯​M 🤯​M 🤯​M ✅​p
🤯​ TestAccept/bundle/deploy/files/no-snapshot-sync 🤯​M ✅​p 🤯​M 🤯​M 🤯​M ✅​p
🤯​ TestAccept/bundle/deploy/files/no-snapshot-sync/DMS= 🤯​M ✅​p 🤯​M 🤯​M 🤯​M ✅​p
🤯​ TestAccept/bundle/deploy/files/no-snapshot-sync/DMS=true 🤯​M ✅​p 🤯​M 🤯​M 🤯​M ✅​p
🤯​ TestAccept/bundle/deploy/immutable/basic 🤯​M ✅​p 🤯​M 🤯​M 🤯​M ✅​p
🤯​ TestAccept/bundle/deploy/immutable/basic/DMS= 🤯​M ✅​p 🤯​M 🤯​M 🤯​M ✅​p
🤯​ TestAccept/bundle/deploy/immutable/breakglass 🤯​M ✅​p 🤯​M 🤯​M 🤯​M ✅​p
🤯​ TestAccept/bundle/deploy/immutable/breakglass/DMS= 🤯​M ✅​p 🤯​M 🤯​M 🤯​M ✅​p
🤯​ TestAccept/bundle/deploy/immutable/can-manage 🤯​M ✅​p 🤯​M 🤯​M 🤯​M ✅​p
🤯​ TestAccept/bundle/deploy/immutable/can-manage/DMS= 🤯​M ✅​p 🤯​M 🤯​M 🤯​M ✅​p
🤯​ TestAccept/bundle/deploy/immutable/destroy-redeploy 🤯​M ✅​p 🤯​M 🤯​M 🤯​M ✅​p
🤯​ TestAccept/bundle/deploy/immutable/destroy-redeploy/DMS= 🤯​M ✅​p 🤯​M 🤯​M 🤯​M ✅​p
🤯​ TestAccept/bundle/deploy/immutable/dev-mode 🤯​M ✅​p 🤯​M 🤯​M 🤯​M ✅​p
🤯​ TestAccept/bundle/deploy/immutable/dev-mode/DMS= 🤯​M ✅​p 🤯​M 🤯​M 🤯​M ✅​p
🤯​ TestAccept/bundle/deploy/immutable/legacy-state 🤯​M ✅​p 🤯​M 🤯​M 🤯​M ✅​p
🤯​ TestAccept/bundle/deploy/immutable/legacy-state/DMS= 🤯​M ✅​p 🤯​M 🤯​M 🤯​M ✅​p
🤯​ TestAccept/bundle/deploy/immutable/missing-snapshot 🤯​M ✅​p 🤯​M 🤯​M 🤯​M ✅​p
🤯​ TestAccept/bundle/deploy/immutable/missing-snapshot/DMS= 🤯​M ✅​p 🤯​M 🤯​M 🤯​M ✅​p
🤯​ TestAccept/bundle/deploy/immutable/no-artifacts 🤯​M ✅​p 🤯​M 🤯​M 🤯​M ✅​p
🤯​ TestAccept/bundle/deploy/immutable/no-artifacts/DMS=/READPLAN= 🤯​M ✅​p 🤯​M 🤯​M 🤯​M ✅​p
🤯​ TestAccept/bundle/deploy/immutable/no-artifacts/DMS=/READPLAN=1 🤯​M ✅​p 🤯​M 🤯​M 🤯​M ✅​p
🤯​ TestAccept/bundle/deploy/immutable/permissions-change 🤯​M ✅​p 🤯​M 🤯​M 🤯​M ✅​p
🤯​ TestAccept/bundle/deploy/immutable/permissions-change/DMS= 🤯​M ✅​p 🤯​M 🤯​M 🤯​M ✅​p
🤯​ TestAccept/bundle/deploy/immutable/select 🤯​M ✅​p 🤯​M 🤯​M 🤯​M ✅​p
🤯​ TestAccept/bundle/deploy/immutable/select/DMS= 🤯​M ✅​p 🤯​M 🤯​M 🤯​M ✅​p
🤯​ TestAccept/bundle/deploy/mlops-stacks 🤯​M ✅​p 🤯​M 🤯​M 🤯​M ✅​p
🤯​ TestAccept/bundle/deploy/mlops-stacks/DMS= 🤯​M ✅​p 🤯​M 🤯​M 🤯​M ✅​p
🤯​ TestAccept/bundle/deploy/mlops-stacks/DMS=true 🤯​M ✅​p 🤯​M 🤯​M 🤯​M ✅​p
🤯​ TestAccept/bundle/deploy/spark-jar-task 🤯​M ✅​p 🤯​M 🤯​M 🤯​M ✅​p
🤯​ TestAccept/bundle/deploy/spark-jar-task/DMS= 🤯​M ✅​p 🤯​M 🤯​M 🤯​M ✅​p
🤯​ TestAccept/bundle/deploy/spark-jar-task/DMS=true 🤯​M ✅​p 🤯​M 🤯​M 🤯​M ✅​p
🤯​ TestAccept/bundle/deployment/bind/alert 🤯​M 💚​R 🤯​M 🤯​M 🤯​M 💚​R
🤯​ TestAccept/bundle/deployment/bind/alert/DMS= 🤯​M 💚​R 🤯​M 🤯​M 🤯​M 💚​R
🤯​ TestAccept/bundle/deployment/bind/alert/DMS=true 🤯​M 💚​R 🤯​M 🤯​M 🤯​M 💚​R
🤯​ TestAccept/bundle/deployment/bind/catalog 🤯​M ✅​p 🤯​M 🤯​M 🤯​M ✅​p
🤯​ TestAccept/bundle/deployment/bind/catalog/DMS= 🤯​M ✅​p 🤯​M 🤯​M 🤯​M ✅​p
🤯​ TestAccept/bundle/deployment/bind/catalog/DMS=true 🤯​M ✅​p 🤯​M 🤯​M 🤯​M ✅​p
🤯​ TestAccept/bundle/deployment/bind/cluster 🤯​M ✅​p 🤯​M 🤯​M ✅​p ✅​p
🤯​ TestAccept/bundle/deployment/bind/cluster/DMS= 🤯​M ✅​p 🤯​M 🤯​M ✅​p ✅​p
🤯​ TestAccept/bundle/deployment/bind/cluster/DMS=true 🤯​M ✅​p 🤯​M 🤯​M ✅​p ✅​p
🤯​ TestAccept/bundle/deployment/bind/dashboard 🤯​M ✅​p 🤯​M 🤯​M 🤯​M ✅​p
🤯​ TestAccept/bundle/deployment/bind/dashboard/DMS=/READPLAN= 🤯​M ✅​p 🤯​M 🤯​M 🤯​M ✅​p
🤯​ TestAccept/bundle/deployment/bind/dashboard/DMS=/READPLAN=1 🤯​M ✅​p 🤯​M 🤯​M 🤯​M ✅​p
🤯​ TestAccept/bundle/deployment/bind/dashboard/DMS=true/READPLAN= 🤯​M ✅​p 🤯​M 🤯​M 🤯​M ✅​p
🤯​ TestAccept/bundle/deployment/bind/dashboard/DMS=true/READPLAN=1 🤯​M ✅​p 🤯​M 🤯​M 🤯​M ✅​p
🤯​ TestAccept/bundle/deployment/bind/dashboard/recreation 🤯​M ✅​p 🤯​M 🤯​M 🤯​M ✅​p
🤯​ TestAccept/bundle/deployment/bind/dashboard/recreation/DMS=/READPLAN= 🤯​M ✅​p 🤯​M 🤯​M 🤯​M ✅​p
🤯​ TestAccept/bundle/deployment/bind/dashboard/recreation/DMS=/READPLAN=1 🤯​M ✅​p 🤯​M 🤯​M 🤯​M ✅​p
Top 50 slowest tests (at least 2 minutes):
duration env testname
20:14 gcp-windows-latest-4core-16gb TestAccept/bundle/resources/apps/lifecycle-started/DMS=true
16:05 gcp-windows-latest-4core-16gb TestAccept/bundle/resources/apps/lifecycle-started/DMS=
15:59 gcp linux-2core-8gb TestAccept/bundle/resources/apps/lifecycle-started/DMS=true
14:44 aws-windows-latest-4core-16gb TestAccept/bundle/resources/apps/lifecycle-started/DMS=
14:35 aws linux-2core-8gb TestAccept/bundle/resources/apps/lifecycle-started/DMS=true
13:32 aws linux-2core-8gb TestAccept/bundle/resources/clusters/lifecycle-started/DMS=true
13:27 aws-windows-latest-4core-16gb TestAccept/bundle/resources/apps/lifecycle-started/DMS=true
13:21 azure linux-2core-8gb TestAccept/bundle/resources/apps/lifecycle-started/DMS=true
12:51 azure linux-2core-8gb TestAccept/bundle/resources/apps/lifecycle-started/DMS=
12:50 aws-windows-latest-4core-16gb TestAccept/bundle/resources/schemas/rename_then_change/DMS=true
12:31 aws linux-2core-8gb TestAccept/bundle/quiet-levels/DMS=
12:31 azure-windows-latest-4core-16gb TestAccept/bundle/resources/apps/lifecycle-started/DMS=true
12:23 aws linux-2core-8gb TestAccept/bundle/resources/apps/lifecycle-started/DMS=
11:58 azure-windows-latest-4core-16gb TestAccept/bundle/resources/apps/lifecycle-started/DMS=
11:34 aws linux-2core-8gb TestAccept/bundle/resources/clusters/lifecycle-started/DMS=
11:34 aws-windows-latest-4core-16gb TestAccept/bundle/resources/clusters/lifecycle-started/DMS=true
11:08 azure linux-2core-8gb TestAccept/bundle/resources/materialization_sets/batch/DMS=true
10:44 azure linux-2core-8gb TestAccept/bundle/resources/materialization_sets/batch/DMS=
10:40 gcp linux-2core-8gb TestAccept/bundle/resources/apps/lifecycle-started/DMS=
10:37 gcp-windows-latest-4core-16gb TestAccept/bundle/resources/apps/lifecycle-started-omitted/DMS=
10:32 gcp linux-2core-8gb TestAccept/bundle/resources/schemas/rename_then_change/DMS=true
10:18 aws linux-2core-8gb TestAccept/bundle/resources/apps/lifecycle-started-toggle/DMS=
10:11 gcp linux-2core-8gb TestAccept/bundle/resources/clusters/lifecycle-started/DMS=
9:50 azure-windows-latest-4core-16gb TestAccept/bundle/resources/clusters/lifecycle-started/DMS=true
9:41 gcp linux-2core-8gb TestAccept/bundle/resources/clusters/lifecycle-started/DMS=true
9:25 azure linux-2core-8gb TestAccept/bundle/resources/clusters/lifecycle-started-toggle/DMS=
9:13 gcp linux-2core-8gb TestAccept/bundle/invariant/no_drift/DMS=true/INPUT_CONFIG=cluster_libraries.yml.tmpl/READPLAN=1
9:09 aws-windows-latest-4core-16gb TestAccept/bundle/resources/clusters/lifecycle-started/DMS=
8:44 azure linux-2core-8gb TestAccept/bundle/resources/clusters/lifecycle-started/DMS=true
8:38 azure-windows-latest-4core-16gb TestAccept/bundle/resources/clusters/lifecycle-started/DMS=
8:29 azure-windows-latest-4core-16gb TestAccept/bundle/resources/schemas/rename_then_change/DMS=
8:29 gcp linux-2core-8gb TestAccept/bundle/resources/schemas/rename_then_change/DMS=
8:25 gcp linux-2core-8gb TestAccept/bundle/templates/default-python/combinations/serverless/DLT=no/DMS=/NBOOK=no/PY=yes/READPLAN=
8:20 aws linux-2core-8gb TestAccept/bundle/resources/apps/lifecycle-started-omitted/DMS=
8:13 gcp linux-2core-8gb TestAccept/bundle/resources/schemas/rename/DMS=true
8:09 azure-windows-latest-4core-16gb TestAccept/bundle/resources/postgres_projects/update_display_name/DMS=true
8:07 azure linux-2core-8gb TestAccept/bundle/quiet-levels/DMS=true
8:06 azure-windows-latest-4core-16gb TestAccept/bundle/resources/schemas/rename_then_change/DMS=true
8:04 gcp linux-2core-8gb TestAccept/bundle/resources/materialization_sets/batch/DMS=true
8:04 gcp linux-2core-8gb TestAccept/bundle/resources/clusters/deploy/local_ssd_count/DMS=
8:01 gcp-windows-latest-4core-16gb TestAccept/bundle/resources/apps/lifecycle-started-omitted/DMS=true
7:59 gcp linux-2core-8gb TestAccept/bundle/resources/clusters/deploy/local_ssd_count/DMS=true
7:58 gcp-windows-latest-4core-16gb TestAccept/bundle/invariant/no_drift/DMS=/INPUT_CONFIG=feature.yml.tmpl/READPLAN=
7:58 azure-windows-latest-4core-16gb TestAccept/bundle/resources/apps/lifecycle-started-toggle/DMS=true
7:52 azure linux-2core-8gb TestAccept/bundle/resources/clusters/lifecycle-started/DMS=
7:48 aws-windows-latest-4core-16gb TestAccept/bundle/resources/materialization_sets/immutable/DMS=
7:47 aws-windows-latest-4core-16gb TestAccept/bundle/apps/compute_size/DMS=
7:43 azure linux-2core-8gb TestAccept/bundle/resources/clusters/lifecycle-started-toggle/DMS=true
7:27 aws linux-2core-8gb TestAccept/bundle/resources/apps/lifecycle-started-omitted/DMS=true
7:27 azure linux-2core-8gb TestAccept/bundle/resources/materialization_sets/immutable/DMS=true
(1125 table rows omitted to keep the report under 60000 bytes)

@denik denik added factory-generated and removed ai-auto Deprecated: use factory-generated instead. labels Nov 20, 2026
idelafuente-oc pushed a commit to idelafuente-oc/cli that referenced this pull request Nov 20, 2026
…databricks#7763)

## Changes

`DbfsClient.Write` restarts the whole DBFS upload (create, add-block,
close) once when add-block or close fails with 404
`RESOURCE_DOES_NOT_EXIST` "No such stream with handle". To restart, it
has to read the source again:
- seekable sources (local files) seek back to where they started;
- non-seekable sources (a download from DBFS or a Volume during `fs cp`,
a pipe) are copied into a memory buffer as they are read, up to 8 MiB.
Past that limit there is no restart, the same limit databricks#7607 uses for
Volumes.

The restart always passes overwrite. The first create succeeded, so the
file at the path now is this upload's own partial file. Without
overwrite, a retried `fs cp` without `--overwrite` would fail on that
file with "already exists". Decision: accept the small window where
another writer creates the same path between the two attempts and gets
overwritten.

The testserver's DBFS fake now returns the same message as the real API
for an unknown handle. I checked the message on AWS: closing a handle
and then calling add-block or close on it returns `404
RESOURCE_DOES_NOT_EXIST "No such stream with handle: [ID]."`.

This is separate from databricks#7607, which changes UC Volumes uploads in
`files_client.go`. The two buffers look alike, but I kept them apart so
the PRs don't depend on each other.

## Why

The backend drops stream handles now and then, a few seconds into
uploads of files that are only a few bytes. It happens on every cloud,
and it failed 38 integration tests (`integration/cmd/fs` and
`integration/libs/filer` DBFS variants) in 21 runs on 2026-11-17,
including main on Azure. Before this change, `DbfsClient.Write` made a
single attempt and the SDK does not retry a 404, so every lost handle
failed the command.

## Dashboard

| error | snippet |
|---|---|
|
[error/1218](http://go/cli/dashboard/error/1218?main=1&pr=1&end=2026-11-18T12:00:00Z&last=7d)
| `dbfs write: No such stream with handle: [ID].` |

Example tests:
[integration/cmd/fs.TestFsCpFileToDirFileNotOverwritten/uc-volumes_to_dbfs](http://go/cli/dashboard/test/integration/cmd/fs.TestFsCpFileToDirFileNotOverwritten/uc-volumes_to_dbfs?main=1&pr=1&end=2026-11-18T12:00:00Z&last=7d),
[integration/cmd/fs.TestFsCpDirToDirWithOverwriteFlag/dbfs_to_dbfs](http://go/cli/dashboard/test/integration/cmd/fs.TestFsCpDirToDirWithOverwriteFlag/dbfs_to_dbfs?main=1&pr=1&end=2026-11-18T12:00:00Z&last=7d)

```
cp_test.go:237:
    Error:      Received unexpected error:
                dbfs write: No such stream with handle: [ID].
    Test:       TestFsCpFileToDirFileNotOverwritten/uc-volumes_to_dbfs
```

## Tests

- New unit tests in `libs/filer/dbfs_client_test.go` use a fake that
drops the handle on add-block or close. They cover non-seekable and
seekable sources, a large source that fails early, the single-restart
limit, the buffer limit, and the case where the error is not a lost
handle. All but the last fail without the fix.
- `TestFsCpFileToDirFileNotOverwritten`,
`TestFsCpDirToDirWithOverwriteFlag` and
`TestFsCpFileToFileWithOverwriteFlag` pass on AWS (30 subtests). A lost
handle could not be triggered against the real backend.

This pull request and its description were written by Isaac.

---------

Co-authored-by: Isaac <no-reply@databricks.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants