Skip to content

fs cp -r: back off during failure bursts - #7786

Draft
yuanjieding-db wants to merge 1 commit into
yuanjieding-db/fs-cp-stall-timeoutfrom
yuanjieding-db/fs-cp-breaker
Draft

yuanjieding-db wants to merge 1 commit into
yuanjieding-db/fs-cp-stall-timeoutfrom
yuanjieding-db/fs-cp-breaker

Conversation

@yuanjieding-db

@yuanjieding-db yuanjieding-db commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

Stack. Each PR is based on the one before it, so its diff shows only its own change.

PR
1 #7711 Skip the per-file parent-directory check in fs cp -r
2 #7784 testserver: optional response headers on fault rules
3 #7712 fs cp -r: keep going past failed files (validation results are here)
4 #7785 fs cp -r: stop when files keep failing for 5 minutes
5 #7786 fs cp -r: back off during failure bursts
6 #7787 fs cp -r: retry transient directory errors

This is part 5.

Changes

After --concurrency consecutive retryable failures with no success in between, a circuit breaker opens:

  • While it's open, no new files start. One deferred file at a time is probed, waiting 2s, 4s, ... up to 30s between probes. A Retry-After hint is honored, capped at 30s.
  • The first success closes the breaker.
  • A probe that fails defers its file again, so during an outage a file can be tried more than twice, up to the stall timeout.
  • Logging: opening and closing the breaker is logged at info, each probe at debug.
  • With --concurrency 1, one retryable failure opens the breaker, so each transient error costs at least a 2s pause before the next file.
  • Probing is serial because admit runs on the walker goroutine (now noted in a comment).

DATABRICKS_FS_CP_PROBE_BACKOFF_MAX_MS is an undocumented override of the 30s cap, for tests. Sleeping is injectable.

Why

Keeping going during a burst of failures sends a failing request for every file and adds to the load on a service that is already struggling.

Tests

  • Unit tests (fake clock):
    • backoff of 2, 4, 8 and 16s, Retry-After above the backoff, and Retry-After capped at 30s, with the probe order and the log lines;
    • the breaker opening during the deferred pass, where the file itself is the probe;
    • opening only after consecutive failures, keeping the largest Retry-After until a success, and closing on success;
    • retryAfter for both SDKs, and the env override.
  • Acceptance:
    • burst-retry-after: six 500s with Retry-After: 1 open the breaker, and every file is copied.
    • outage-past-stall: uploads fail for longer than a 2s stall timeout, and the copy stops and reports. This is fs cp -r: stop when files keep failing for 5 minutes #7785's end-to-end scenario: it needs the breaker's waits to reach the timeout.
    • The burst uses 500s, not 503s, because the filer retries a 503 itself for about 75 seconds before the copy sees it. The 123K-file validation covers the real 503 path (see fs cp -r: keep going past failed files #7712).

Self-review

Isaac review, 1 finding:

  • Fixed: the probe where the job itself is the probe had no test. TestRecursiveCopyBreakerOnDeferredPass covers it now.

/code-review (medium), 8 findings:

  • Fixed: a deferred file can be uploaded more than twice while the breaker probes. That's how the copy retries during an outage, bounded by the stall timeout, and the struct doc now says so.
  • Not changed:
    • A probe can be a large file. This change is about many small files.
    • A DBFS Retry-After in the HTTP-date form is ignored. The DBFS SDK retries 429 and 503 itself, and the seconds form is honored.
    • The backoff doubles even when a sleep is cut short. That only happens when the copy is stopping.
    • The largest Retry-After is held until a success. That's intentional: a later 500 without a hint must not erase a 503's hint.
    • --concurrency 1 opens the breaker on one failure. The spec sets the threshold to --concurrency.
    • admit returns one bool for both "stopped" and "copied as the probe". Both mean "don't start it", admit records a not-started file itself, and it has one caller.
    • Copies started before the breaker opened keep running. They were already sent, and cancelling them would waste the work.
    • The outage unit test now covers one deferred file instead of five. The case where many deferred files keep their errors at a stall is covered in fs cp -r: stop when files keep failing for 5 minutes #7785 and fs cp -r: keep going past failed files #7712.

This pull request and its description were written by Isaac.

@yuanjieding-db
yuanjieding-db force-pushed the yuanjieding-db/fs-cp-stall-timeout branch from 6d97669 to 85e902d Compare November 18, 2026 18:14
@yuanjieding-db
yuanjieding-db force-pushed the yuanjieding-db/fs-cp-breaker branch from af4903e to 717bb8b Compare November 18, 2026 18:14
@github-actions

github-actions Bot commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

Commit: 54d80127b | Workflow run

Regression test report

Selected 6 changed tests (limit=20, 0 not selected)

6 tests: regression=6 unreleased=0 coverage=0 failing=0
test branch main (4e503ea) latest (v1.26.0)
TestAccept/cmd/fs/cp-r-keep-going/burst-retry-after ✅ ❌ ❌
TestAccept/cmd/fs/cp-r-keep-going/outage-past-stall ✅ ❌ ❌
TestAccept/cmd/fs/cp-r-keep-going/persistent-500 ✅ ❌ ❌
TestAccept/cmd/fs/cp-r-keep-going/single-500 ✅ ❌ ❌
TestAccept/cmd/fs/cp-r-requests ✅ ❌ ❌
TestAccept/cmd/fs/cp/dir-to-dir ✅ ❌ ❌
TestAccept/cmd/fs/cp-r-keep-going/burst-retry-after: branch ✅ | main (4e503ea) ❌ | latest (v1.26.0) ❌

main (4e503ea):

acceptance_test.go:1380: 
        	Error Trace:	/home/runner/work/cli/cli/libs/testdiff/testdiff.go:22
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:1380
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:1121
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:666
        	Error:      	Not equal: 
        	            	expected: "\n>>> cp_r localdir dbfs:/Volumes/main/default/keep-going/dst --concurrency 2\nCopied 6 files.\nlocaldir/f1.txt -> dbfs:/Volumes/main/default/keep-going/dst/f1.txt\nlocaldir/f2.txt -> dbfs:/Volumes/main/default/keep-going/dst/f2.txt\nlocaldir/f3.txt -> dbfs:/Volumes/main/default/keep-going/dst/f3.txt\nlocaldir/f4.txt -> dbfs:/Volumes/main/default/keep-going/dst/f4.txt\nlocaldir/f5.txt -> dbfs:/Volumes/main/default/keep-going/dst/f5.txt\nlocaldir/f6.txt -> dbfs:/Volumes/main/default/keep-going/dst/f6.txt\n\n=== Six uploads plus the six that hit the burst\n\n>>> print_requests.py --summary //api/2.0/fs/files/\n12\n"
        	            	actual  : "\n>>> cp_r localdir dbfs:/Volumes/main/default/keep-going/dst --concurrency 2\nError: databricks-api error: Fault injected by test.\n\nExit code: 1\n"
        	            	
        	            	Diff:
        	            	--- Expected
        	            	+++ Actual
        	            	@@ -2,14 +2,5 @@
        	            	 >>> cp_r localdir dbfs:/Volumes/main/default/keep-going/dst --concurrency 2
        	            	-Copied 6 files.
        	            	-localdir/f1.txt -> dbfs:/Volumes/main/default/keep-going/dst/f1.txt
        	            	-localdir/f2.txt -> dbfs:/Volumes/main/default/keep-going/dst/f2.txt
        	            	-localdir/f3.txt -> dbfs:/Volumes/main/default/keep-going/dst/f3.txt
        	            	-localdir/f4.txt -> dbfs:/Volumes/main/default/keep-going/dst/f4.txt
        	            	-localdir/f5.txt -> dbfs:/Volumes/main/default/keep-going/dst/f5.txt
        	            	-localdir/f6.txt -> dbfs:/Volumes/main/default/keep-going/dst/f6.txt
        	            	+Error: databricks-api error: Fault injected by test.
        	            	 
        	            	-=== Six uploads plus the six that hit the burst
        	            	+Exit code: 1
        	            	 
        	            	->>> print_requests.py --summary //api/2.0/fs/files/
        	            	-12
        	            	-
        	Test:       	TestAccept/cmd/fs/cp-r-keep-going/burst-retry-after
        	Messages:   	cmd/fs/cp-r-keep-going/burst-retry-after/output.txt vs /tmp/TestAcceptcmdfscp-r-keep-goingburst-retry-after53586011/002/output.txt

latest (v1.26.0):

server.go:104: Error while reading request body: unexpected EOF
    acceptance_test.go:1380: 
        	Error Trace:	/home/runner/work/cli/cli/libs/testdiff/testdiff.go:22
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:1380
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:1121
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:666
        	Error:      	Not equal: 
        	            	expected: "\n>>> cp_r localdir dbfs:/Volumes/main/default/keep-going/dst --concurrency 2\nCopied 6 files.\nlocaldir/f1.txt -> dbfs:/Volumes/main/default/keep-going/dst/f1.txt\nlocaldir/f2.txt -> dbfs:/Volumes/main/default/keep-going/dst/f2.txt\nlocaldir/f3.txt -> dbfs:/Volumes/main/default/keep-going/dst/f3.txt\nlocaldir/f4.txt -> dbfs:/Volumes/main/default/keep-going/dst/f4.txt\nlocaldir/f5.txt -> dbfs:/Volumes/main/default/keep-going/dst/f5.txt\nlocaldir/f6.txt -> dbfs:/Volumes/main/default/keep-going/dst/f6.txt\n\n=== Six uploads plus the six that hit the burst\n\n>>> print_requests.py --summary //api/2.0/fs/files/\n12\n"
        	            	actual  : "\n>>> cp_r localdir dbfs:/Volumes/main/default/keep-going/dst --concurrency 2\nError: databricks-api error: Fault injected by test.\n\nExit code: 1\n"
        	            	
        	            	Diff:
        	            	--- Expected
        	            	+++ Actual
        	            	@@ -2,14 +2,5 @@
        	            	 >>> cp_r localdir dbfs:/Volumes/main/default/keep-going/dst --concurrency 2
        	            	-Copied 6 files.
        	            	-localdir/f1.txt -> dbfs:/Volumes/main/default/keep-going/dst/f1.txt
        	            	-localdir/f2.txt -> dbfs:/Volumes/main/default/keep-going/dst/f2.txt
        	            	-localdir/f3.txt -> dbfs:/Volumes/main/default/keep-going/dst/f3.txt
        	            	-localdir/f4.txt -> dbfs:/Volumes/main/default/keep-going/dst/f4.txt
        	            	-localdir/f5.txt -> dbfs:/Volumes/main/default/keep-going/dst/f5.txt
        	            	-localdir/f6.txt -> dbfs:/Volumes/main/default/keep-going/dst/f6.txt
        	            	+Error: databricks-api error: Fault injected by test.
        	            	 
        	            	-=== Six uploads plus the six that hit the burst
        	            	+Exit code: 1
        	            	 
        	            	->>> print_requests.py --summary //api/2.0/fs/files/
        	            	-12
        	            	-
        	Test:       	TestAccept/cmd/fs/cp-r-keep-going/burst-retry-after
        	Messages:   	cmd/fs/cp-r-keep-going/burst-retry-after/output.txt vs /tmp/TestAcceptcmdfscp-r-keep-goingburst-retry-after682687113/002/output.txt
TestAccept/cmd/fs/cp-r-keep-going/outage-past-stall: branch ✅ | main (4e503ea) ❌ | latest (v1.26.0) ❌

main (4e503ea):

acceptance_test.go:1380: 
        	Error Trace:	/home/runner/work/cli/cli/libs/testdiff/testdiff.go:22
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:1380
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:1121
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:666
        	Error:      	Not equal: 
        	            	expected: "\n>>> cp_r localdir dbfs:/Volumes/main/default/keep-going/dst --concurrency 1\nCopy incomplete: 0 copied, 0 skipped (already existed), 1 failed, 3 not attempted.\nStopped: files kept failing and none was copied for 2 seconds, so no new files were started.\nFailed:\n  localdir/f1.txt -> dbfs:/Volumes/main/default/keep-going/dst/f1.txt: databricks-api error: Fault injected by test. (HTTP 500 Internal Server Error)\nError: copy stopped: files kept failing and none was copied for 2 seconds\n\nExit code: 1\n"
        	            	actual  : "\n>>> cp_r localdir dbfs:/Volumes/main/default/keep-going/dst --concurrency 1\nError: databricks-api error: Fault injected by test.\n\nExit code: 1\n"
        	            	
        	            	Diff:
        	            	--- Expected
        	            	+++ Actual
        	            	@@ -2,7 +2,3 @@
        	            	 >>> cp_r localdir dbfs:/Volumes/main/default/keep-going/dst --concurrency 1
        	            	-Copy incomplete: 0 copied, 0 skipped (already existed), 1 failed, 3 not attempted.
        	            	-Stopped: files kept failing and none was copied for 2 seconds, so no new files were started.
        	            	-Failed:
        	            	-  localdir/f1.txt -> dbfs:/Volumes/main/default/keep-going/dst/f1.txt: databricks-api error: Fault injected by test. (HTTP 500 Internal Server Error)
        	            	-Error: copy stopped: files kept failing and none was copied for 2 seconds
        	            	+Error: databricks-api error: Fault injected by test.
        	            	 
        	Test:       	TestAccept/cmd/fs/cp-r-keep-going/outage-past-stall
        	Messages:   	cmd/fs/cp-r-keep-going/outage-past-stall/output.txt vs /tmp/TestAcceptcmdfscp-r-keep-goingoutage-past-stall1616184824/002/output.txt

latest (v1.26.0):

acceptance_test.go:1380: 
        	Error Trace:	/home/runner/work/cli/cli/libs/testdiff/testdiff.go:22
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:1380
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:1121
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:666
        	Error:      	Not equal: 
        	            	expected: "\n>>> cp_r localdir dbfs:/Volumes/main/default/keep-going/dst --concurrency 1\nCopy incomplete: 0 copied, 0 skipped (already existed), 1 failed, 3 not attempted.\nStopped: files kept failing and none was copied for 2 seconds, so no new files were started.\nFailed:\n  localdir/f1.txt -> dbfs:/Volumes/main/default/keep-going/dst/f1.txt: databricks-api error: Fault injected by test. (HTTP 500 Internal Server Error)\nError: copy stopped: files kept failing and none was copied for 2 seconds\n\nExit code: 1\n"
        	            	actual  : "\n>>> cp_r localdir dbfs:/Volumes/main/default/keep-going/dst --concurrency 1\nError: databricks-api error: Fault injected by test.\n\nExit code: 1\n"
        	            	
        	            	Diff:
        	            	--- Expected
        	            	+++ Actual
        	            	@@ -2,7 +2,3 @@
        	            	 >>> cp_r localdir dbfs:/Volumes/main/default/keep-going/dst --concurrency 1
        	            	-Copy incomplete: 0 copied, 0 skipped (already existed), 1 failed, 3 not attempted.
        	            	-Stopped: files kept failing and none was copied for 2 seconds, so no new files were started.
        	            	-Failed:
        	            	-  localdir/f1.txt -> dbfs:/Volumes/main/default/keep-going/dst/f1.txt: databricks-api error: Fault injected by test. (HTTP 500 Internal Server Error)
        	            	-Error: copy stopped: files kept failing and none was copied for 2 seconds
        	            	+Error: databricks-api error: Fault injected by test.
        	            	 
        	Test:       	TestAccept/cmd/fs/cp-r-keep-going/outage-past-stall
        	Messages:   	cmd/fs/cp-r-keep-going/outage-past-stall/output.txt vs /tmp/TestAcceptcmdfscp-r-keep-goingoutage-past-stall2588497379/002/output.txt
FAIL
TestAccept/cmd/fs/cp-r-keep-going/persistent-500: branch ✅ | main (4e503ea) ❌ | latest (v1.26.0) ❌

main (4e503ea):

acceptance_test.go:1380: Diff:
        --- cmd/fs/cp-r-keep-going/persistent-500/output.txt
        +++ /tmp/TestAcceptcmdfscp-r-keep-goingpersistent-5003459153444/002/output.txt
        @@ -1,12 +1,8 @@
         
         >>> cp_r localdir dbfs:/Volumes/main/default/keep-going/dst --concurrency 2
        -Copy incomplete: 3 copied, 0 skipped (already existed), 1 failed, 0 not attempted.
        -Failed:
        -  localdir/f3.txt -> dbfs:/Volumes/main/default/keep-going/dst/f3.txt: HTTP 500 Internal Server Error
        -Error: 1 file was not copied
        +Error: databricks-api error: 
         localdir/f1.txt -> dbfs:/Volumes/main/default/keep-going/dst/f1.txt
         localdir/f2.txt -> dbfs:/Volumes/main/default/keep-going/dst/f2.txt
        -localdir/f4.txt -> dbfs:/Volumes/main/default/keep-going/dst/f4.txt
         
         Exit code: 1
         
        @@ -16,5 +10,3 @@
         PUT /api/2.0/fs/files/Volumes/main/default/keep-going/dst/f1.txt overwrite=false
         PUT /api/2.0/fs/files/Volumes/main/default/keep-going/dst/f2.txt overwrite=false
         PUT /api/2.0/fs/files/Volumes/main/default/keep-going/dst/f3.txt overwrite=false
        -PUT /api/2.0/fs/files/Volumes/main/default/keep-going/dst/f3.txt overwrite=false
        -PUT /api/2.0/fs/files/Volumes/main/default/keep-going/dst/f4.txt overwrite=false

latest (v1.26.0):

acceptance_test.go:1380: Diff:
        --- cmd/fs/cp-r-keep-going/persistent-500/output.txt
        +++ /tmp/TestAcceptcmdfscp-r-keep-goingpersistent-5004133525395/002/output.txt
        @@ -1,12 +1,8 @@
         
         >>> cp_r localdir dbfs:/Volumes/main/default/keep-going/dst --concurrency 2
        -Copy incomplete: 3 copied, 0 skipped (already existed), 1 failed, 0 not attempted.
        -Failed:
        -  localdir/f3.txt -> dbfs:/Volumes/main/default/keep-going/dst/f3.txt: HTTP 500 Internal Server Error
        -Error: 1 file was not copied
        +Error: databricks-api error: 
         localdir/f1.txt -> dbfs:/Volumes/main/default/keep-going/dst/f1.txt
         localdir/f2.txt -> dbfs:/Volumes/main/default/keep-going/dst/f2.txt
        -localdir/f4.txt -> dbfs:/Volumes/main/default/keep-going/dst/f4.txt
         
         Exit code: 1
         
        @@ -16,5 +10,4 @@
         PUT /api/2.0/fs/files/Volumes/main/default/keep-going/dst/f1.txt overwrite=false
         PUT /api/2.0/fs/files/Volumes/main/default/keep-going/dst/f2.txt overwrite=false
         PUT /api/2.0/fs/files/Volumes/main/default/keep-going/dst/f3.txt overwrite=false
        -PUT /api/2.0/fs/files/Volumes/main/default/keep-going/dst/f3.txt overwrite=false
         PUT /api/2.0/fs/files/Volumes/main/default/keep-going/dst/f4.txt overwrite=false
TestAccept/cmd/fs/cp-r-keep-going/single-500: branch ✅ | main (4e503ea) ❌ | latest (v1.26.0) ❌

main (4e503ea):

acceptance_test.go:1380: Diff:
        --- cmd/fs/cp-r-keep-going/single-500/output.txt
        +++ /tmp/TestAcceptcmdfscp-r-keep-goingsingle-500209949259/002/output.txt
        @@ -1,20 +1,8 @@
         
         >>> cp_r localdir dbfs:/Volumes/main/default/keep-going/dst --concurrency 2
        -Copied 6 files.
        +Error: databricks-api error: Fault injected by test.
         localdir/f1.txt -> dbfs:/Volumes/main/default/keep-going/dst/f1.txt
         localdir/f2.txt -> dbfs:/Volumes/main/default/keep-going/dst/f2.txt
        -localdir/f3.txt -> dbfs:/Volumes/main/default/keep-going/dst/f3.txt
         localdir/f4.txt -> dbfs:/Volumes/main/default/keep-going/dst/f4.txt
        -localdir/f5.txt -> dbfs:/Volumes/main/default/keep-going/dst/f5.txt
        -localdir/f6.txt -> dbfs:/Volumes/main/default/keep-going/dst/f6.txt
         
        -=== Six uploads plus one retry
        -
        ->>> print_requests.py --summary --sort //api/2.0/fs/files/
        -PUT /api/2.0/fs/files/Volumes/main/default/keep-going/dst/f1.txt overwrite=false
        -PUT /api/2.0/fs/files/Volumes/main/default/keep-going/dst/f2.txt overwrite=false
        -PUT /api/2.0/fs/files/Volumes/main/default/keep-going/dst/f3.txt overwrite=false
        -PUT /api/2.0/fs/files/Volumes/main/default/keep-going/dst/f3.txt overwrite=false
        -PUT /api/2.0/fs/files/Volumes/main/default/keep-going/dst/f4.txt overwrite=false
        -PUT /api/2.0/fs/files/Volumes/main/default/keep-going/dst/f5.txt overwrite=false
        -PUT /api/2.0/fs/files/Volumes/main/default/keep-going/dst/f6.txt overwrite=false
        +Exit code: 1

latest (v1.26.0):

acceptance_test.go:1380: Diff:
        --- cmd/fs/cp-r-keep-going/single-500/output.txt
        +++ /tmp/TestAcceptcmdfscp-r-keep-goingsingle-500454255148/002/output.txt
        @@ -1,20 +1,7 @@
         
         >>> cp_r localdir dbfs:/Volumes/main/default/keep-going/dst --concurrency 2
        -Copied 6 files.
        +Error: databricks-api error: Fault injected by test.
         localdir/f1.txt -> dbfs:/Volumes/main/default/keep-going/dst/f1.txt
         localdir/f2.txt -> dbfs:/Volumes/main/default/keep-going/dst/f2.txt
        -localdir/f3.txt -> dbfs:/Volumes/main/default/keep-going/dst/f3.txt
        -localdir/f4.txt -> dbfs:/Volumes/main/default/keep-going/dst/f4.txt
        -localdir/f5.txt -> dbfs:/Volumes/main/default/keep-going/dst/f5.txt
        -localdir/f6.txt -> dbfs:/Volumes/main/default/keep-going/dst/f6.txt
         
        -=== Six uploads plus one retry
        -
        ->>> print_requests.py --summary --sort //api/2.0/fs/files/
        -PUT /api/2.0/fs/files/Volumes/main/default/keep-going/dst/f1.txt overwrite=false
        -PUT /api/2.0/fs/files/Volumes/main/default/keep-going/dst/f2.txt overwrite=false
        -PUT /api/2.0/fs/files/Volumes/main/default/keep-going/dst/f3.txt overwrite=false
        -PUT /api/2.0/fs/files/Volumes/main/default/keep-going/dst/f3.txt overwrite=false
        -PUT /api/2.0/fs/files/Volumes/main/default/keep-going/dst/f4.txt overwrite=false
        -PUT /api/2.0/fs/files/Volumes/main/default/keep-going/dst/f5.txt overwrite=false
        -PUT /api/2.0/fs/files/Volumes/main/default/keep-going/dst/f6.txt overwrite=false
        +Exit code: 1
TestAccept/cmd/fs/cp-r-requests: branch ✅ | main (4e503ea) ❌ | latest (v1.26.0) ❌

main (4e503ea):

acceptance_test.go:1380: Diff:
        --- cmd/fs/cp-r-requests/output.txt
        +++ /tmp/TestAcceptcmdfscp-r-requests231917254/002/output.txt
        @@ -1,6 +1,5 @@
         
         >>> [CLI] fs cp -r localdir dbfs:/Volumes/main/default/fs-cp-r-requests/dst
        -Copied 3 files.
         localdir/a.txt -> dbfs:/Volumes/main/default/fs-cp-r-requests/dst/a.txt
         localdir/b.txt -> dbfs:/Volumes/main/default/fs-cp-r-requests/dst/b.txt
         localdir/sub/c.txt -> dbfs:/Volumes/main/default/fs-cp-r-requests/dst/sub/c.txt
        @@ -8,6 +7,9 @@
         === Recursive copy: a directory PUT per directory, a file PUT per file, no directory HEAD per file
         
         >>> print_requests.py --sort //api/2.0/fs/
        +{"method":"HEAD","path":"/api/2.0/fs/directories/Volumes/main/default/fs-cp-r-requests/dst"}
        +{"method":"HEAD","path":"/api/2.0/fs/directories/Volumes/main/default/fs-cp-r-requests/dst"}
        +{"method":"HEAD","path":"/api/2.0/fs/directories/Volumes/main/default/fs-cp-r-requests/dst/sub"}
         {"method":"PUT","path":"/api/2.0/fs/directories/Volumes/main/default/fs-cp-r-requests/dst"}
         {"method":"PUT","path":"/api/2.0/fs/directories/Volumes/main/default/fs-cp-r-requests/dst/sub"}
         {"method":"PUT","path":"/api/2.0/fs/files/Volumes/main/default/fs-cp-r-requests/dst/a.txt"}
        
FAIL

latest (v1.26.0):

acceptance_test.go:1380: Diff:
        --- cmd/fs/cp-r-requests/output.txt
        +++ /tmp/TestAcceptcmdfscp-r-requests1824640186/002/output.txt
        @@ -1,6 +1,5 @@
         
         >>> [CLI] fs cp -r localdir dbfs:/Volumes/main/default/fs-cp-r-requests/dst
        -Copied 3 files.
         localdir/a.txt -> dbfs:/Volumes/main/default/fs-cp-r-requests/dst/a.txt
         localdir/b.txt -> dbfs:/Volumes/main/default/fs-cp-r-requests/dst/b.txt
         localdir/sub/c.txt -> dbfs:/Volumes/main/default/fs-cp-r-requests/dst/sub/c.txt
        @@ -8,6 +7,9 @@
         === Recursive copy: a directory PUT per directory, a file PUT per file, no directory HEAD per file
         
         >>> print_requests.py --sort //api/2.0/fs/
        +{"method":"HEAD","path":"/api/2.0/fs/directories/Volumes/main/default/fs-cp-r-requests/dst"}
        +{"method":"HEAD","path":"/api/2.0/fs/directories/Volumes/main/default/fs-cp-r-requests/dst"}
        +{"method":"HEAD","path":"/api/2.0/fs/directories/Volumes/main/default/fs-cp-r-requests/dst/sub"}
         {"method":"PUT","path":"/api/2.0/fs/directories/Volumes/main/default/fs-cp-r-requests/dst"}
         {"method":"PUT","path":"/api/2.0/fs/directories/Volumes/main/default/fs-cp-r-requests/dst/sub"}
         {"method":"PUT","path":"/api/2.0/fs/files/Volumes/main/default/fs-cp-r-requests/dst/a.txt"}
TestAccept/cmd/fs/cp/dir-to-dir: branch ✅ | main (4e503ea) ❌ | latest (v1.26.0) ❌

main (4e503ea):

acceptance_test.go:1380: 
        	Error Trace:	/home/runner/work/cli/cli/libs/testdiff/testdiff.go:22
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:1380
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:1121
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:666
        	Error:      	Not equal: 
        	            	expected: "\n>>> [CLI] fs cp -r localdir dbfs:/Volumes/main/default/fs-cp-test-[UNIQUE_NAME]/uploaded-dir\nCopied 2 files.\nlocaldir/file1.txt -> dbfs:/Volumes/main/default/fs-cp-test-[UNIQUE_NAME]/uploaded-dir/file1.txt\nlocaldir/file2.txt -> dbfs:/Volumes/main/default/fs-cp-test-[UNIQUE_NAME]/uploaded-dir/file2.txt\n\n>>> [CLI] fs cat dbfs:/Volumes/main/default/fs-cp-test-[UNIQUE_NAME]/uploaded-dir/file1.txt\nfile1 content\n\n>>> [CLI] fs cat dbfs:/Volumes/main/default/fs-cp-test-[UNIQUE_NAME]/uploaded-dir/file2.txt\nfile2 content\n"
        	            	actual  : "\n>>> [CLI] fs cp -r localdir dbfs:/Volumes/main/default/fs-cp-test-[UNIQUE_NAME]/uploaded-dir\nlocaldir/file1.txt -> dbfs:/Volumes/main/default/fs-cp-test-[UNIQUE_NAME]/uploaded-dir/file1.txt\nlocaldir/file2.txt -> dbfs:/Volumes/main/default/fs-cp-test-[UNIQUE_NAME]/uploaded-dir/file2.txt\n\n>>> [CLI] fs cat dbfs:/Volumes/main/default/fs-cp-test-[UNIQUE_NAME]/uploaded-dir/file1.txt\nfile1 content\n\n>>> [CLI] fs cat dbfs:/Volumes/main/default/fs-cp-test-[UNIQUE_NAME]/uploaded-dir/file2.txt\nfile2 content\n"
        	            	
        	            	Diff:
        	            	--- Expected
        	            	+++ Actual
        	            	@@ -2,3 +2,2 @@
        	            	 >>> [CLI] fs cp -r localdir dbfs:/Volumes/main/default/fs-cp-test-[UNIQUE_NAME]/uploaded-dir
        	            	-Copied 2 files.
        	            	 localdir/file1.txt -> dbfs:/Volumes/main/default/fs-cp-test-[UNIQUE_NAME]/uploaded-dir/file1.txt
        	Test:       	TestAccept/cmd/fs/cp/dir-to-dir
        	Messages:   	cmd/fs/cp/dir-to-dir/output.txt vs /tmp/TestAcceptcmdfscpdir-to-dir1132527176/002/output.txt

latest (v1.26.0): same output as main (4e503ea)

@eng-dev-ecosystem-bot

eng-dev-ecosystem-bot commented Nov 18, 2026 •

Copy link
Copy Markdown
Collaborator

✅ Integration test report

Commit: 54d8012

Run: 44387730329

Env 🔄​flaky 💚​RECOVERED ✅​pass 🙈​skip Time
💚​ aws linux-2core-8gb 1 271 20 10:07
💚​ aws-windows-latest-4core-16gb 1 273 18 4:47
💚​ azure linux-2core-8gb 1 270 20 9:57
🔄​ azure-windows-latest-4core-16gb 1 272 18 6:59
💚​ gcp linux-2core-8gb 1 271 20 11:09
💚​ gcp-windows-latest-4core-16gb 1 273 18 5:23
Test Name aws linux-2core-8gb aws-windows-latest-4core-16gb azure linux-2core-8gb azure-windows-latest-4core-16gb gcp linux-2core-8gb gcp-windows-latest-4core-16gb
🔄​ TestAccept 💚​R 💚​R 💚​R 🔄​f 💚​R 💚​R
Top 6 slowest tests (at least 2 minutes):
duration env testname
3:55 gcp linux-2core-8gb TestAccept
3:51 azure linux-2core-8gb TestAccept
3:46 aws linux-2core-8gb TestAccept
3:01 gcp linux-2core-8gb TestFilerRecursiveDelete/workspace_files
2:37 gcp-windows-latest-4core-16gb TestFilerWorkspaceNotebook/scalaJupyterNb.ipynb
2:17 azure-windows-latest-4core-16gb TestImportDirWithOverwriteFlag

Keeping going during a burst of failures sends a failing request for
every file. After --concurrency consecutive retryable failures with no
success in between, a circuit breaker now opens: no new files start, and
one deferred file at a time is probed, waiting 2s, 4s, ... up to 30s
between probes. A Retry-After hint is honored, capped at 30s. The first
success closes the breaker. Opening and closing it is logged at info,
each probe at debug.

DATABRICKS_FS_CP_PROBE_BACKOFF_MAX_MS is an undocumented test-only
override of the 30s cap. Sleeping is injectable, and unit tests use a
fake clock.

The acceptance tests cover a 500 burst with Retry-After, and an outage
longer than the stall timeout, which needs the breaker's waits to reach
the timeout.

Co-authored-by: Isaac <no-reply@databricks.com>
@yuanjieding-db
yuanjieding-db force-pushed the yuanjieding-db/fs-cp-stall-timeout branch from cd40cbd to 5dbeb76 Compare November 18, 2026 20:05
@yuanjieding-db
yuanjieding-db force-pushed the yuanjieding-db/fs-cp-breaker branch from 4a552ce to 54d8012 Compare November 18, 2026 20:05

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants