Skip to content

fs cp -r: retry transient directory errors - #7787

Draft
yuanjieding-db wants to merge 1 commit into
yuanjieding-db/fs-cp-breakerfrom
yuanjieding-db/fs-cp-walk-retry
Draft

yuanjieding-db wants to merge 1 commit into
yuanjieding-db/fs-cp-breakerfrom
yuanjieding-db/fs-cp-walk-retry

Conversation

@yuanjieding-db

@yuanjieding-db yuanjieding-db commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

Stack. Each PR is based on the one before it, so its diff shows only its own change.

PR
1 #7711 Skip the per-file parent-directory check in fs cp -r
2 #7784 testserver: optional response headers on fault rules
3 #7712 fs cp -r: keep going past failed files (validation results are here)
4 #7785 fs cp -r: stop when files keep failing for 5 minutes
5 #7786 fs cp -r: back off during failure bursts
6 #7787 fs cp -r: retry transient directory errors

This is part 6.

Changes

A 500 on one Mkdir or source listing used to end the whole copy. A retryable error on a target Mkdir, a source listing, or the walk's Stat of the source root is now retried in place:

  • Pace: one call at a time, with the breaker's 2s to 30s backoff and Retry-After.
  • Limit: up to the stall timeout from the first attempt. A call that recovers counts as progress, so its retries don't use up the files' stall timeout.
  • Still fatal: a non-retryable error, or one that outlasts the timeout, still stops the copy, and the summary names the error.
  • While stalled: a stalled copy that is only counting files doesn't retry.

Why

The goal is a large copy that finishes despite transient 5xx errors. A directory call is one of the requests that can hit one.

Tests

  • TestRecursiveCopyWalkRetry (fake clock):
    • a Mkdir 500 is retried in place;
    • a listing that recovers after 4m30s of retries leaves the next file its stall timeout;
    • a listing that fails for the whole stall timeout stops the copy, and the summary names its error;
    • a stalled copy doesn't retry listings (the only sleep is the breaker's).
  • Acceptance: cmd/fs/cp-r-keep-going/mkdir-500.

Self-review

Isaac review, 1 finding:

  • Fixed: the retry window was measured from the last success, and could mark the copy stalled. It now runs from the first attempt, and retrying checks only for cancellation or an existing stall.

/code-review (medium), 9 findings:

  • Fixed:
    • A directory that recovered could use up the stall budget. A recovered call counts as progress now, with a test.
    • The summary blamed a stall over the directory error. It names the error now.
    • The walk's Stat of the source root wasn't retried. It's retried again, as in the validated build.
    • A test row asserted no log line. It checks one now.
  • Not changed:
    • The time each call takes isn't bounded. Every call has the filer's own retries, so the release note says "about 5 minutes".
    • Retries are logged at debug. That matches the per-file deferral lines; info is for the breaker and the stall.
    • This adds a second retry layer. The filer's retry policy is out of scope for this stack, and this only covers what gets past it.
    • The backoff calculation appears twice. It's two lines, and the breaker keeps its state across calls while the walk retry doesn't.
    • Acceptance covers only a Mkdir 500. Listing retries and DBFS errors have unit tests.

This pull request and its description were written by Isaac.

@yuanjieding-db
yuanjieding-db force-pushed the yuanjieding-db/fs-cp-breaker branch from af4903e to 717bb8b Compare November 18, 2026 18:14
@yuanjieding-db
yuanjieding-db force-pushed the yuanjieding-db/fs-cp-walk-retry branch from a7c5b35 to e704d85 Compare November 18, 2026 18:14
@github-actions

github-actions Bot commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

Commit: 9b1f622d8 | Workflow run

Regression test report

Selected 7 changed tests (limit=20, 0 not selected)

7 tests: regression=7 unreleased=0 coverage=0 failing=0
test branch main (4e503ea) latest (v1.26.0)
TestAccept/cmd/fs/cp-r-keep-going/burst-retry-after ✅ ❌ ❌
TestAccept/cmd/fs/cp-r-keep-going/mkdir-500 ✅ ❌ ❌
TestAccept/cmd/fs/cp-r-keep-going/outage-past-stall ✅ ❌ ❌
TestAccept/cmd/fs/cp-r-keep-going/persistent-500 ✅ ❌ ❌
TestAccept/cmd/fs/cp-r-keep-going/single-500 ✅ ❌ ❌
TestAccept/cmd/fs/cp-r-requests ✅ ❌ ❌
TestAccept/cmd/fs/cp/dir-to-dir ✅ ❌ ❌
TestAccept/cmd/fs/cp-r-keep-going/burst-retry-after: branch ✅ | main (4e503ea) ❌ | latest (v1.26.0) ❌

main (4e503ea):

acceptance_test.go:1380: 
        	Error Trace:	/home/runner/work/cli/cli/libs/testdiff/testdiff.go:22
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:1380
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:1121
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:666
        	Error:      	Not equal: 
        	            	expected: "\n>>> cp_r localdir dbfs:/Volumes/main/default/keep-going/dst --concurrency 2\nCopied 6 files.\nlocaldir/f1.txt -> dbfs:/Volumes/main/default/keep-going/dst/f1.txt\nlocaldir/f2.txt -> dbfs:/Volumes/main/default/keep-going/dst/f2.txt\nlocaldir/f3.txt -> dbfs:/Volumes/main/default/keep-going/dst/f3.txt\nlocaldir/f4.txt -> dbfs:/Volumes/main/default/keep-going/dst/f4.txt\nlocaldir/f5.txt -> dbfs:/Volumes/main/default/keep-going/dst/f5.txt\nlocaldir/f6.txt -> dbfs:/Volumes/main/default/keep-going/dst/f6.txt\n\n=== Six uploads plus the six that hit the burst\n\n>>> print_requests.py --summary //api/2.0/fs/files/\n12\n"
        	            	actual  : "\n>>> cp_r localdir dbfs:/Volumes/main/default/keep-going/dst --concurrency 2\nError: databricks-api error: Fault injected by test.\n\nExit code: 1\n"
        	            	
        	            	Diff:
        	            	--- Expected
        	            	+++ Actual
        	            	@@ -2,14 +2,5 @@
        	            	 >>> cp_r localdir dbfs:/Volumes/main/default/keep-going/dst --concurrency 2
        	            	-Copied 6 files.
        	            	-localdir/f1.txt -> dbfs:/Volumes/main/default/keep-going/dst/f1.txt
        	            	-localdir/f2.txt -> dbfs:/Volumes/main/default/keep-going/dst/f2.txt
        	            	-localdir/f3.txt -> dbfs:/Volumes/main/default/keep-going/dst/f3.txt
        	            	-localdir/f4.txt -> dbfs:/Volumes/main/default/keep-going/dst/f4.txt
        	            	-localdir/f5.txt -> dbfs:/Volumes/main/default/keep-going/dst/f5.txt
        	            	-localdir/f6.txt -> dbfs:/Volumes/main/default/keep-going/dst/f6.txt
        	            	+Error: databricks-api error: Fault injected by test.
        	            	 
        	            	-=== Six uploads plus the six that hit the burst
        	            	+Exit code: 1
        	            	 
        	            	->>> print_requests.py --summary //api/2.0/fs/files/
        	            	-12
        	            	-
        	Test:       	TestAccept/cmd/fs/cp-r-keep-going/burst-retry-after
        	Messages:   	cmd/fs/cp-r-keep-going/burst-retry-after/output.txt vs /tmp/TestAcceptcmdfscp-r-keep-goingburst-retry-after1070278390/002/output.txt

latest (v1.26.0): same output as main (4e503ea)

TestAccept/cmd/fs/cp-r-keep-going/mkdir-500: branch ✅ | main (4e503ea) ❌ | latest (v1.26.0) ❌

main (4e503ea):

acceptance_test.go:1380: 
        	Error Trace:	/home/runner/work/cli/cli/libs/testdiff/testdiff.go:22
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:1380
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:1121
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:666
        	Error:      	Not equal: 
        	            	expected: "\n>>> cp_r localdir dbfs:/Volumes/main/default/keep-going/dst --concurrency 2\nCopied 3 files.\nlocaldir/f1.txt -> dbfs:/Volumes/main/default/keep-going/dst/f1.txt\nlocaldir/f2.txt -> dbfs:/Volumes/main/default/keep-going/dst/f2.txt\nlocaldir/sub/f.txt -> dbfs:/Volumes/main/default/keep-going/dst/sub/f.txt\n\n=== The PUT of dst/sub is sent twice\n\n>>> print_requests.py --summary --sort //api/2.0/fs/directories/\nPUT /api/2.0/fs/directories/Volumes/main/default/keep-going/dst\nPUT /api/2.0/fs/directories/Volumes/main/default/keep-going/dst/sub\nPUT /api/2.0/fs/directories/Volumes/main/default/keep-going/dst/sub\n"
        	            	actual  : "\n>>> cp_r localdir dbfs:/Volumes/main/default/keep-going/dst --concurrency 2\nError: databricks-api error: Fault injected by test.\n\nExit code: 1\n"
        	            	
        	            	Diff:
        	            	--- Expected
        	            	+++ Actual
        	            	@@ -2,13 +2,5 @@
        	            	 >>> cp_r localdir dbfs:/Volumes/main/default/keep-going/dst --concurrency 2
        	            	-Copied 3 files.
        	            	-localdir/f1.txt -> dbfs:/Volumes/main/default/keep-going/dst/f1.txt
        	            	-localdir/f2.txt -> dbfs:/Volumes/main/default/keep-going/dst/f2.txt
        	            	-localdir/sub/f.txt -> dbfs:/Volumes/main/default/keep-going/dst/sub/f.txt
        	            	+Error: databricks-api error: Fault injected by test.
        	            	 
        	            	-=== The PUT of dst/sub is sent twice
        	            	+Exit code: 1
        	            	 
        	            	->>> print_requests.py --summary --sort //api/2.0/fs/directories/
        	            	-PUT /api/2.0/fs/directories/Volumes/main/default/keep-going/dst
        	            	-PUT /api/2.0/fs/directories/Volumes/main/default/keep-going/dst/sub
        	            	-PUT /api/2.0/fs/directories/Volumes/main/default/keep-going/dst/sub
        	            	-
        	Test:       	TestAccept/cmd/fs/cp-r-keep-going/mkdir-500
        	Messages:   	cmd/fs/cp-r-keep-going/mkdir-500/output.txt vs /tmp/TestAcceptcmdfscp-r-keep-goingmkdir-500282526363/002/output.txt
FAIL

latest (v1.26.0):

acceptance_test.go:1380: 
        	Error Trace:	/home/runner/work/cli/cli/libs/testdiff/testdiff.go:22
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:1380
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:1121
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:666
        	Error:      	Not equal: 
        	            	expected: "\n>>> cp_r localdir dbfs:/Volumes/main/default/keep-going/dst --concurrency 2\nCopied 3 files.\nlocaldir/f1.txt -> dbfs:/Volumes/main/default/keep-going/dst/f1.txt\nlocaldir/f2.txt -> dbfs:/Volumes/main/default/keep-going/dst/f2.txt\nlocaldir/sub/f.txt -> dbfs:/Volumes/main/default/keep-going/dst/sub/f.txt\n\n=== The PUT of dst/sub is sent twice\n\n>>> print_requests.py --summary --sort //api/2.0/fs/directories/\nPUT /api/2.0/fs/directories/Volumes/main/default/keep-going/dst\nPUT /api/2.0/fs/directories/Volumes/main/default/keep-going/dst/sub\nPUT /api/2.0/fs/directories/Volumes/main/default/keep-going/dst/sub\n"
        	            	actual  : "\n>>> cp_r localdir dbfs:/Volumes/main/default/keep-going/dst --concurrency 2\nError: databricks-api error: Fault injected by test.\n\nExit code: 1\n"
        	            	
        	            	Diff:
        	            	--- Expected
        	            	+++ Actual
        	            	@@ -2,13 +2,5 @@
        	            	 >>> cp_r localdir dbfs:/Volumes/main/default/keep-going/dst --concurrency 2
        	            	-Copied 3 files.
        	            	-localdir/f1.txt -> dbfs:/Volumes/main/default/keep-going/dst/f1.txt
        	            	-localdir/f2.txt -> dbfs:/Volumes/main/default/keep-going/dst/f2.txt
        	            	-localdir/sub/f.txt -> dbfs:/Volumes/main/default/keep-going/dst/sub/f.txt
        	            	+Error: databricks-api error: Fault injected by test.
        	            	 
        	            	-=== The PUT of dst/sub is sent twice
        	            	+Exit code: 1
        	            	 
        	            	->>> print_requests.py --summary --sort //api/2.0/fs/directories/
        	            	-PUT /api/2.0/fs/directories/Volumes/main/default/keep-going/dst
        	            	-PUT /api/2.0/fs/directories/Volumes/main/default/keep-going/dst/sub
        	            	-PUT /api/2.0/fs/directories/Volumes/main/default/keep-going/dst/sub
        	            	-
        	Test:       	TestAccept/cmd/fs/cp-r-keep-going/mkdir-500
        	Messages:   	cmd/fs/cp-r-keep-going/mkdir-500/output.txt vs /tmp/TestAcceptcmdfscp-r-keep-goingmkdir-5003883738891/002/output.txt
TestAccept/cmd/fs/cp-r-keep-going/outage-past-stall: branch ✅ | main (4e503ea) ❌ | latest (v1.26.0) ❌

main (4e503ea):

acceptance_test.go:1380: 
        	Error Trace:	/home/runner/work/cli/cli/libs/testdiff/testdiff.go:22
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:1380
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:1121
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:666
        	Error:      	Not equal: 
        	            	expected: "\n>>> cp_r localdir dbfs:/Volumes/main/default/keep-going/dst --concurrency 1\nCopy incomplete: 0 copied, 0 skipped (already existed), 1 failed, 3 not attempted.\nStopped: files kept failing and none was copied for 2 seconds, so no new files were started.\nFailed:\n  localdir/f1.txt -> dbfs:/Volumes/main/default/keep-going/dst/f1.txt: databricks-api error: Fault injected by test. (HTTP 500 Internal Server Error)\nError: copy stopped: files kept failing and none was copied for 2 seconds\n\nExit code: 1\n"
        	            	actual  : "\n>>> cp_r localdir dbfs:/Volumes/main/default/keep-going/dst --concurrency 1\nError: databricks-api error: Fault injected by test.\n\nExit code: 1\n"
        	            	
        	            	Diff:
        	            	--- Expected
        	            	+++ Actual
        	            	@@ -2,7 +2,3 @@
        	            	 >>> cp_r localdir dbfs:/Volumes/main/default/keep-going/dst --concurrency 1
        	            	-Copy incomplete: 0 copied, 0 skipped (already existed), 1 failed, 3 not attempted.
        	            	-Stopped: files kept failing and none was copied for 2 seconds, so no new files were started.
        	            	-Failed:
        	            	-  localdir/f1.txt -> dbfs:/Volumes/main/default/keep-going/dst/f1.txt: databricks-api error: Fault injected by test. (HTTP 500 Internal Server Error)
        	            	-Error: copy stopped: files kept failing and none was copied for 2 seconds
        	            	+Error: databricks-api error: Fault injected by test.
        	            	 
        	Test:       	TestAccept/cmd/fs/cp-r-keep-going/outage-past-stall
        	Messages:   	cmd/fs/cp-r-keep-going/outage-past-stall/output.txt vs /tmp/TestAcceptcmdfscp-r-keep-goingoutage-past-stall2193047952/002/output.txt

latest (v1.26.0):

acceptance_test.go:1380: 
        	Error Trace:	/home/runner/work/cli/cli/libs/testdiff/testdiff.go:22
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:1380
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:1121
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:666
        	Error:      	Not equal: 
        	            	expected: "\n>>> cp_r localdir dbfs:/Volumes/main/default/keep-going/dst --concurrency 1\nCopy incomplete: 0 copied, 0 skipped (already existed), 1 failed, 3 not attempted.\nStopped: files kept failing and none was copied for 2 seconds, so no new files were started.\nFailed:\n  localdir/f1.txt -> dbfs:/Volumes/main/default/keep-going/dst/f1.txt: databricks-api error: Fault injected by test. (HTTP 500 Internal Server Error)\nError: copy stopped: files kept failing and none was copied for 2 seconds\n\nExit code: 1\n"
        	            	actual  : "\n>>> cp_r localdir dbfs:/Volumes/main/default/keep-going/dst --concurrency 1\nError: databricks-api error: Fault injected by test.\n\nExit code: 1\n"
        	            	
        	            	Diff:
        	            	--- Expected
        	            	+++ Actual
        	            	@@ -2,7 +2,3 @@
        	            	 >>> cp_r localdir dbfs:/Volumes/main/default/keep-going/dst --concurrency 1
        	            	-Copy incomplete: 0 copied, 0 skipped (already existed), 1 failed, 3 not attempted.
        	            	-Stopped: files kept failing and none was copied for 2 seconds, so no new files were started.
        	            	-Failed:
        	            	-  localdir/f1.txt -> dbfs:/Volumes/main/default/keep-going/dst/f1.txt: databricks-api error: Fault injected by test. (HTTP 500 Internal Server Error)
        	            	-Error: copy stopped: files kept failing and none was copied for 2 seconds
        	            	+Error: databricks-api error: Fault injected by test.
        	            	 
        	Test:       	TestAccept/cmd/fs/cp-r-keep-going/outage-past-stall
        	Messages:   	cmd/fs/cp-r-keep-going/outage-past-stall/output.txt vs /tmp/TestAcceptcmdfscp-r-keep-goingoutage-past-stall4202315385/002/output.txt
FAIL
TestAccept/cmd/fs/cp-r-keep-going/persistent-500: branch ✅ | main (4e503ea) ❌ | latest (v1.26.0) ❌

main (4e503ea):

server.go:104: Error while reading request body: unexpected EOF
    acceptance_test.go:1380: Diff:
        --- cmd/fs/cp-r-keep-going/persistent-500/output.txt
        +++ /tmp/TestAcceptcmdfscp-r-keep-goingpersistent-5003666021293/002/output.txt
        @@ -1,12 +1,8 @@
         
         >>> cp_r localdir dbfs:/Volumes/main/default/keep-going/dst --concurrency 2
        -Copy incomplete: 3 copied, 0 skipped (already existed), 1 failed, 0 not attempted.
        -Failed:
        -  localdir/f3.txt -> dbfs:/Volumes/main/default/keep-going/dst/f3.txt: HTTP 500 Internal Server Error
        -Error: 1 file was not copied
        +Error: databricks-api error: 
         localdir/f1.txt -> dbfs:/Volumes/main/default/keep-going/dst/f1.txt
         localdir/f2.txt -> dbfs:/Volumes/main/default/keep-going/dst/f2.txt
        -localdir/f4.txt -> dbfs:/Volumes/main/default/keep-going/dst/f4.txt
         
         Exit code: 1
         
        @@ -16,5 +10,4 @@
         PUT /api/2.0/fs/files/Volumes/main/default/keep-going/dst/f1.txt overwrite=false
         PUT /api/2.0/fs/files/Volumes/main/default/keep-going/dst/f2.txt overwrite=false
         PUT /api/2.0/fs/files/Volumes/main/default/keep-going/dst/f3.txt overwrite=false
        -PUT /api/2.0/fs/files/Volumes/main/default/keep-going/dst/f3.txt overwrite=false
         PUT /api/2.0/fs/files/Volumes/main/default/keep-going/dst/f4.txt overwrite=false

latest (v1.26.0):

acceptance_test.go:1380: Diff:
        --- cmd/fs/cp-r-keep-going/persistent-500/output.txt
        +++ /tmp/TestAcceptcmdfscp-r-keep-goingpersistent-500184108529/002/output.txt
        @@ -1,12 +1,8 @@
         
         >>> cp_r localdir dbfs:/Volumes/main/default/keep-going/dst --concurrency 2
        -Copy incomplete: 3 copied, 0 skipped (already existed), 1 failed, 0 not attempted.
        -Failed:
        -  localdir/f3.txt -> dbfs:/Volumes/main/default/keep-going/dst/f3.txt: HTTP 500 Internal Server Error
        -Error: 1 file was not copied
        +Error: databricks-api error: 
         localdir/f1.txt -> dbfs:/Volumes/main/default/keep-going/dst/f1.txt
         localdir/f2.txt -> dbfs:/Volumes/main/default/keep-going/dst/f2.txt
        -localdir/f4.txt -> dbfs:/Volumes/main/default/keep-going/dst/f4.txt
         
         Exit code: 1
         
        @@ -16,5 +10,4 @@
         PUT /api/2.0/fs/files/Volumes/main/default/keep-going/dst/f1.txt overwrite=false
         PUT /api/2.0/fs/files/Volumes/main/default/keep-going/dst/f2.txt overwrite=false
         PUT /api/2.0/fs/files/Volumes/main/default/keep-going/dst/f3.txt overwrite=false
        -PUT /api/2.0/fs/files/Volumes/main/default/keep-going/dst/f3.txt overwrite=false
         PUT /api/2.0/fs/files/Volumes/main/default/keep-going/dst/f4.txt overwrite=false
TestAccept/cmd/fs/cp-r-keep-going/single-500: branch ✅ | main (4e503ea) ❌ | latest (v1.26.0) ❌

main (4e503ea):

acceptance_test.go:1380: Diff:
        --- cmd/fs/cp-r-keep-going/single-500/output.txt
        +++ /tmp/TestAcceptcmdfscp-r-keep-goingsingle-5002800287036/002/output.txt
        @@ -1,20 +1,6 @@
         
         >>> cp_r localdir dbfs:/Volumes/main/default/keep-going/dst --concurrency 2
        -Copied 6 files.
        +Error: databricks-api error: Fault injected by test.
         localdir/f1.txt -> dbfs:/Volumes/main/default/keep-going/dst/f1.txt
        -localdir/f2.txt -> dbfs:/Volumes/main/default/keep-going/dst/f2.txt
        -localdir/f3.txt -> dbfs:/Volumes/main/default/keep-going/dst/f3.txt
        -localdir/f4.txt -> dbfs:/Volumes/main/default/keep-going/dst/f4.txt
        -localdir/f5.txt -> dbfs:/Volumes/main/default/keep-going/dst/f5.txt
        -localdir/f6.txt -> dbfs:/Volumes/main/default/keep-going/dst/f6.txt
         
        -=== Six uploads plus one retry
        -
        ->>> print_requests.py --summary --sort //api/2.0/fs/files/
        -PUT /api/2.0/fs/files/Volumes/main/default/keep-going/dst/f1.txt overwrite=false
        -PUT /api/2.0/fs/files/Volumes/main/default/keep-going/dst/f2.txt overwrite=false
        -PUT /api/2.0/fs/files/Volumes/main/default/keep-going/dst/f3.txt overwrite=false
        -PUT /api/2.0/fs/files/Volumes/main/default/keep-going/dst/f3.txt overwrite=false
        -PUT /api/2.0/fs/files/Volumes/main/default/keep-going/dst/f4.txt overwrite=false
        -PUT /api/2.0/fs/files/Volumes/main/default/keep-going/dst/f5.txt overwrite=false
        -PUT /api/2.0/fs/files/Volumes/main/default/keep-going/dst/f6.txt overwrite=false
        +Exit code: 1

latest (v1.26.0):

acceptance_test.go:1380: Diff:
        --- cmd/fs/cp-r-keep-going/single-500/output.txt
        +++ /tmp/TestAcceptcmdfscp-r-keep-goingsingle-500608617902/002/output.txt
        @@ -1,20 +1,8 @@
         
         >>> cp_r localdir dbfs:/Volumes/main/default/keep-going/dst --concurrency 2
        -Copied 6 files.
        +Error: databricks-api error: Fault injected by test.
         localdir/f1.txt -> dbfs:/Volumes/main/default/keep-going/dst/f1.txt
         localdir/f2.txt -> dbfs:/Volumes/main/default/keep-going/dst/f2.txt
        -localdir/f3.txt -> dbfs:/Volumes/main/default/keep-going/dst/f3.txt
         localdir/f4.txt -> dbfs:/Volumes/main/default/keep-going/dst/f4.txt
        -localdir/f5.txt -> dbfs:/Volumes/main/default/keep-going/dst/f5.txt
        -localdir/f6.txt -> dbfs:/Volumes/main/default/keep-going/dst/f6.txt
         
        -=== Six uploads plus one retry
        -
        ->>> print_requests.py --summary --sort //api/2.0/fs/files/
        -PUT /api/2.0/fs/files/Volumes/main/default/keep-going/dst/f1.txt overwrite=false
        -PUT /api/2.0/fs/files/Volumes/main/default/keep-going/dst/f2.txt overwrite=false
        -PUT /api/2.0/fs/files/Volumes/main/default/keep-going/dst/f3.txt overwrite=false
        -PUT /api/2.0/fs/files/Volumes/main/default/keep-going/dst/f3.txt overwrite=false
        -PUT /api/2.0/fs/files/Volumes/main/default/keep-going/dst/f4.txt overwrite=false
        -PUT /api/2.0/fs/files/Volumes/main/default/keep-going/dst/f5.txt overwrite=false
        -PUT /api/2.0/fs/files/Volumes/main/default/keep-going/dst/f6.txt overwrite=false
        +Exit code: 1
TestAccept/cmd/fs/cp-r-requests: branch ✅ | main (4e503ea) ❌ | latest (v1.26.0) ❌

main (4e503ea):

acceptance_test.go:1380: Diff:
        --- cmd/fs/cp-r-requests/output.txt
        +++ /tmp/TestAcceptcmdfscp-r-requests1870165414/002/output.txt
        @@ -1,6 +1,5 @@
         
         >>> [CLI] fs cp -r localdir dbfs:/Volumes/main/default/fs-cp-r-requests/dst
        -Copied 3 files.
         localdir/a.txt -> dbfs:/Volumes/main/default/fs-cp-r-requests/dst/a.txt
         localdir/b.txt -> dbfs:/Volumes/main/default/fs-cp-r-requests/dst/b.txt
         localdir/sub/c.txt -> dbfs:/Volumes/main/default/fs-cp-r-requests/dst/sub/c.txt
        @@ -8,6 +7,9 @@
         === Recursive copy: a directory PUT per directory, a file PUT per file, no directory HEAD per file
         
         >>> print_requests.py --sort //api/2.0/fs/
        +{"method":"HEAD","path":"/api/2.0/fs/directories/Volumes/main/default/fs-cp-r-requests/dst"}
        +{"method":"HEAD","path":"/api/2.0/fs/directories/Volumes/main/default/fs-cp-r-requests/dst"}
        +{"method":"HEAD","path":"/api/2.0/fs/directories/Volumes/main/default/fs-cp-r-requests/dst/sub"}
         {"method":"PUT","path":"/api/2.0/fs/directories/Volumes/main/default/fs-cp-r-requests/dst"}
         {"method":"PUT","path":"/api/2.0/fs/directories/Volumes/main/default/fs-cp-r-requests/dst/sub"}
         {"method":"PUT","path":"/api/2.0/fs/files/Volumes/main/default/fs-cp-r-requests/dst/a.txt"}

latest (v1.26.0): same output as main (4e503ea)

TestAccept/cmd/fs/cp/dir-to-dir: branch ✅ | main (4e503ea) ❌ | latest (v1.26.0) ❌

main (4e503ea):

acceptance_test.go:1380: 
        	Error Trace:	/home/runner/work/cli/cli/libs/testdiff/testdiff.go:22
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:1380
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:1121
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:666
        	Error:      	Not equal: 
        	            	expected: "\n>>> [CLI] fs cp -r localdir dbfs:/Volumes/main/default/fs-cp-test-[UNIQUE_NAME]/uploaded-dir\nCopied 2 files.\nlocaldir/file1.txt -> dbfs:/Volumes/main/default/fs-cp-test-[UNIQUE_NAME]/uploaded-dir/file1.txt\nlocaldir/file2.txt -> dbfs:/Volumes/main/default/fs-cp-test-[UNIQUE_NAME]/uploaded-dir/file2.txt\n\n>>> [CLI] fs cat dbfs:/Volumes/main/default/fs-cp-test-[UNIQUE_NAME]/uploaded-dir/file1.txt\nfile1 content\n\n>>> [CLI] fs cat dbfs:/Volumes/main/default/fs-cp-test-[UNIQUE_NAME]/uploaded-dir/file2.txt\nfile2 content\n"
        	            	actual  : "\n>>> [CLI] fs cp -r localdir dbfs:/Volumes/main/default/fs-cp-test-[UNIQUE_NAME]/uploaded-dir\nlocaldir/file1.txt -> dbfs:/Volumes/main/default/fs-cp-test-[UNIQUE_NAME]/uploaded-dir/file1.txt\nlocaldir/file2.txt -> dbfs:/Volumes/main/default/fs-cp-test-[UNIQUE_NAME]/uploaded-dir/file2.txt\n\n>>> [CLI] fs cat dbfs:/Volumes/main/default/fs-cp-test-[UNIQUE_NAME]/uploaded-dir/file1.txt\nfile1 content\n\n>>> [CLI] fs cat dbfs:/Volumes/main/default/fs-cp-test-[UNIQUE_NAME]/uploaded-dir/file2.txt\nfile2 content\n"
        	            	
        	            	Diff:
        	            	--- Expected
        	            	+++ Actual
        	            	@@ -2,3 +2,2 @@
        	            	 >>> [CLI] fs cp -r localdir dbfs:/Volumes/main/default/fs-cp-test-[UNIQUE_NAME]/uploaded-dir
        	            	-Copied 2 files.
        	            	 localdir/file1.txt -> dbfs:/Volumes/main/default/fs-cp-test-[UNIQUE_NAME]/uploaded-dir/file1.txt
        	Test:       	TestAccept/cmd/fs/cp/dir-to-dir
        	Messages:   	cmd/fs/cp/dir-to-dir/output.txt vs /tmp/TestAcceptcmdfscpdir-to-dir1865603470/002/output.txt

latest (v1.26.0): same output as main (4e503ea)

@eng-dev-ecosystem-bot

eng-dev-ecosystem-bot commented Nov 18, 2026 •

Copy link
Copy Markdown
Collaborator

✅ Integration test report

Commit: 9b1f622

Run: 44387766702

Env 🔄​flaky 💚​RECOVERED ✅​pass 🙈​skip Time
💚​ aws linux-2core-8gb 1 271 21 10:31
💚​ aws-windows-latest-4core-16gb 1 273 19 6:26
💚​ azure linux-2core-8gb 1 270 21 11:58
🔄​ azure-windows-latest-4core-16gb 1 1 271 19 7:01
💚​ gcp linux-2core-8gb 1 271 21 11:54
💚​ gcp-windows-latest-4core-16gb 1 273 19 6:26
Test Name aws linux-2core-8gb aws-windows-latest-4core-16gb azure linux-2core-8gb azure-windows-latest-4core-16gb gcp linux-2core-8gb gcp-windows-latest-4core-16gb
💚​ TestAccept 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R
🔄​ TestSyncFullFileSync ✅​p ✅​p ✅​p 🔄​f ✅​p ✅​p
Top 21 slowest tests (at least 2 minutes):
duration env testname
5:08 aws linux-2core-8gb TestFsCpSourceIsDirectoryButTargetIsFile/uc-volumes_to_local
5:06 azure linux-2core-8gb TestFsCpSourceIsDirectoryButTargetIsFile/uc-volumes_to_local
5:06 aws-windows-latest-4core-16gb TestFsCpSourceIsDirectoryButTargetIsFile/dbfs_to_local
5:04 gcp linux-2core-8gb TestFsCpSourceIsDirectoryButTargetIsFile/dbfs_to_local
5:04 aws-windows-latest-4core-16gb TestFsCpSourceIsDirectoryButTargetIsFile/uc-volumes_to_local
5:04 azure-windows-latest-4core-16gb TestFsCpSourceIsDirectoryButTargetIsFile/uc-volumes_to_local
5:03 gcp linux-2core-8gb TestFsCpSourceIsDirectoryButTargetIsFile/uc-volumes_to_local
5:03 gcp-windows-latest-4core-16gb TestFsCpSourceIsDirectoryButTargetIsFile/uc-volumes_to_local
5:03 gcp-windows-latest-4core-16gb TestFsCpSourceIsDirectoryButTargetIsFile/dbfs_to_local
5:03 aws linux-2core-8gb TestFsCpSourceIsDirectoryButTargetIsFile/dbfs_to_local
5:02 azure-windows-latest-4core-16gb TestFsCpSourceIsDirectoryButTargetIsFile/dbfs_to_local
5:02 azure linux-2core-8gb TestFsCpSourceIsDirectoryButTargetIsFile/dbfs_to_local
5:00 azure-windows-latest-4core-16gb TestFsCpSourceIsDirectoryButTargetIsFile/local_to_local
5:00 azure linux-2core-8gb TestFsCpSourceIsDirectoryButTargetIsFile/local_to_local
5:00 gcp-windows-latest-4core-16gb TestFsCpSourceIsDirectoryButTargetIsFile/local_to_local
5:00 aws linux-2core-8gb TestFsCpSourceIsDirectoryButTargetIsFile/local_to_local
5:00 aws-windows-latest-4core-16gb TestFsCpSourceIsDirectoryButTargetIsFile/local_to_local
5:00 gcp linux-2core-8gb TestFsCpSourceIsDirectoryButTargetIsFile/local_to_local
3:59 gcp linux-2core-8gb TestAccept
3:56 aws linux-2core-8gb TestAccept
3:49 azure linux-2core-8gb TestAccept

A 500 on one Mkdir or source listing used to end the whole copy. A
retryable error on a target Mkdir or a source directory listing during
the walk is now retried in place, one call at a time, with the breaker's
2s to 30s backoff and Retry-After, until the call has failed for the
stall timeout. A non-retryable error, or one that outlasts the timeout,
still stops the copy.

Co-authored-by: Isaac <no-reply@databricks.com>
@yuanjieding-db
yuanjieding-db force-pushed the yuanjieding-db/fs-cp-breaker branch from 4a552ce to 54d8012 Compare November 18, 2026 20:05
@yuanjieding-db
yuanjieding-db force-pushed the yuanjieding-db/fs-cp-walk-retry branch from a7cefac to 9b1f622 Compare November 18, 2026 20:06

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants