Skip to content

fs cp -r: stop when files keep failing for 5 minutes - #7785

Draft
yuanjieding-db wants to merge 1 commit into
yuanjieding-db/fs-cp-keep-goingfrom
yuanjieding-db/fs-cp-stall-timeout
Draft

yuanjieding-db wants to merge 1 commit into
yuanjieding-db/fs-cp-keep-goingfrom
yuanjieding-db/fs-cp-stall-timeout

Conversation

@yuanjieding-db

@yuanjieding-db yuanjieding-db commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

Stack. Each PR is based on the one before it, so its diff shows only its own change.

PR
1 #7711 Skip the per-file parent-directory check in fs cp -r
2 #7784 testserver: optional response headers on fault rules
3 #7712 fs cp -r: keep going past failed files (validation results are here)
4 #7785 fs cp -r: stop when files keep failing for 5 minutes
5 #7786 fs cp -r: back off during failure bursts
6 #7787 fs cp -r: retry transient directory errors

This is part 4.

Changes

If files keep failing and none has been copied for 5 minutes, fs cp -r now starts no new files, lets in-flight ones finish, and reports the rest as not attempted. Files that already failed once keep their error.

  • Only a copy failing with retryable errors can stall: 408, 429, 5xx, or network errors, the kind an outage causes. The timeout runs from the later of the first such failure since the last file success and the last progress, where a created directory also counts as progress.
  • What doesn't stall: a slow copy with no failures, such as large files or a slow listing. Nor a copy that fails once after a long healthy stretch, or that hits a permanent error such as a 404.
  • Visible output: the stop is logged at info, and the summary says Stopped: files kept failing and none was copied for 5 minutes.

The timeout is internal, not a flag. DATABRICKS_FS_CP_STALL_TIMEOUT_MS is an undocumented override for tests. The clock is injectable.

Why

During an outage every file fails, so keeping going would only work through the rest of the copy one failure at a time.

Tests

  • TestRecursiveCopyStall (fake clock):
    • an outage stops the copy, and no file starts once files have been failing for the timeout;
    • a success resets the timer;
    • slow directories count as progress;
    • a slow copy with no failures never stalls (a 6-minute listing and 6-minute writes);
    • a listing error while the stalled copy is only counting files keeps the stall as the reason.
  • TestStallTimesTheFailingPeriod: one failure after 6 healthy minutes with nothing completing doesn't stall; 5 minutes of failing does.
  • TestPermanentErrorsDoNotStall: a 404 followed by a 6-minute listing doesn't stall.
  • TestStallTimeoutFromEnv and TestFormatDuration.
  • No acceptance scenario here. The end-to-end scenario, an outage longer than the timeout, lands in fs cp -r: back off during failure bursts #7786. Without the breaker's waits, an all-500 outage fails every file in milliseconds and never reaches a stall timeout.
  • On kartik-test: three 200 MiB files at --concurrency 1 with a 5s timeout copy fine (see fs cp -r: keep going past failed files #7712).

Self-review

Isaac review, 5 findings:

  • Fixed:
    • Directory work didn't count as progress. A successful Mkdir now resets the timer, with a test.
    • The walk-error-after-stall branch had no test. That branch is gone: while the walk only counts files, a failing listing just ends the count. That case has a test now.
  • Not changed:
    • A stalled copy keeps listing the source to count the files left. The count is useful, and those listings get no retries (fs cp -r: retry transient directory errors #7787 makes sure of that).
    • A hung request blocks the check. The filer's request timeouts bound that, and it's out of scope here.
    • An in-progress upload doesn't count as progress. That's by design for this many-small-files change. A single file that takes more than 5 minutes and then fails stops the copy. Measuring progress in bytes could be a follow-up.

/code-review (medium), 8 findings:

  • Fixed:
    • The summary's reason and the returned error could name different causes. They now use the same order.
    • A walk error while the walk was only counting files replaced the stall error. It's ignored now.
  • Not changed:
    • A slow large file stalls the copy (same as above).
    • The stall is sticky. That's the spec: stop starting files.
    • The clock starts at run. Listing is interleaved with copying, so the first files start right away.
    • The check takes a lock. That's negligible next to the one HTTP request each file costs.
    • durationFromEnv resembles the retry interval parsing in libs/retry. That one has different semantics, and this helper is 6 lines.
    • The env override isn't driven end to end here. fs cp -r: back off during failure bursts #7786's acceptance test uses it.

Orchestrator review:

  • Fixed: the stall fired on slow but healthy copies. It now needs a failure.
  • Checked: the reported case, three 200 MiB files at --concurrency 1 with a 5s timeout, didn't reproduce on kartik-test before the fix: it exited 0 in 67s. The stall is only checked when a file starts, which happens right after a copy frees a slot. The fix still closes the rarer cases, like a slow listing with nothing in flight, and makes the rule explicit. After the fix, the same check exits 0 in 70s.

Isaac review, second and third rounds: no findings.

/code-review (medium), second round, 10 findings:

  • Fixed:
    • The timer ran from the last success, so one failure after a long healthy stretch stalled at once. It now runs from the first failure.
    • The summary said "no file was copied in 5 minutes" even when in-flight files succeeded after the stop. It now says "files kept failing and none was copied for 5 minutes", describing why it stopped.
    • The info log ran under the lock. It now runs after unlock.
  • Not changed:
    • The stall is sticky, and a slow listing doesn't count as progress (see above).
    • Empty directories after the stop aren't created. The summary already says the copy is incomplete.
    • A walk error while counting is ignored. It's now logged at debug.
    • The env override and the duration format are test-only.
    • Progress and success are kept separate on purpose: only a file success clears the failing state.

/code-review (medium), third round, 9 findings:

  • Fixed:
    • Any failure, including a permanent 404, started the clock. Now only retryable errors do.
    • A dropped walk error left no trace. It's logged at debug.
    • The tests depended on the timing env vars in the process environment. The harness pins them now.
  • Not changed:
    • A successful Mkdir resets the timer during a write-only outage. During an outage the breaker holds the walk in place, so no new directories are created.
    • The check isn't timer-driven, and the summary wording (see above).
    • A failed print counts as a failure. It stops the copy anyway.
    • The env override can overflow, and duplicates libs/retry. It's test-only, as above.

This pull request and its description were written by Isaac.

@yuanjieding-db
yuanjieding-db force-pushed the yuanjieding-db/fs-cp-stall-timeout branch from 6d97669 to 85e902d Compare November 18, 2026 18:14
@github-actions

github-actions Bot commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

Commit: 5dbeb76dc | Workflow run

Regression test report

Selected 4 changed tests (limit=20, 0 not selected)

4 tests: regression=4 unreleased=0 coverage=0 failing=0
test branch main (4e503ea) latest (v1.26.0)
TestAccept/cmd/fs/cp-r-keep-going/persistent-500 ✅ ❌ ❌
TestAccept/cmd/fs/cp-r-keep-going/single-500 ✅ ❌ ❌
TestAccept/cmd/fs/cp-r-requests ✅ ❌ ❌
TestAccept/cmd/fs/cp/dir-to-dir ✅ ❌ ❌
TestAccept/cmd/fs/cp-r-keep-going/persistent-500: branch ✅ | main (4e503ea) ❌ | latest (v1.26.0) ❌

main (4e503ea):

acceptance_test.go:1380: Diff:
        --- cmd/fs/cp-r-keep-going/persistent-500/output.txt
        +++ /tmp/TestAcceptcmdfscp-r-keep-goingpersistent-500903941478/002/output.txt
        @@ -1,9 +1,6 @@
         
         >>> cp_r localdir dbfs:/Volumes/main/default/keep-going/dst --concurrency 2
        -Copy incomplete: 3 copied, 0 skipped (already existed), 1 failed, 0 not attempted.
        -Failed:
        -  localdir/f3.txt -> dbfs:/Volumes/main/default/keep-going/dst/f3.txt: HTTP 500 Internal Server Error
        -Error: 1 file was not copied
        +Error: databricks-api error: 
         localdir/f1.txt -> dbfs:/Volumes/main/default/keep-going/dst/f1.txt
         localdir/f2.txt -> dbfs:/Volumes/main/default/keep-going/dst/f2.txt
         localdir/f4.txt -> dbfs:/Volumes/main/default/keep-going/dst/f4.txt
        @@ -16,5 +13,4 @@
         PUT /api/2.0/fs/files/Volumes/main/default/keep-going/dst/f1.txt overwrite=false
         PUT /api/2.0/fs/files/Volumes/main/default/keep-going/dst/f2.txt overwrite=false
         PUT /api/2.0/fs/files/Volumes/main/default/keep-going/dst/f3.txt overwrite=false
        -PUT /api/2.0/fs/files/Volumes/main/default/keep-going/dst/f3.txt overwrite=false
         PUT /api/2.0/fs/files/Volumes/main/default/keep-going/dst/f4.txt overwrite=false

latest (v1.26.0):

server.go:104: Error while reading request body: unexpected EOF
    acceptance_test.go:1380: Diff:
        --- cmd/fs/cp-r-keep-going/persistent-500/output.txt
        +++ /tmp/TestAcceptcmdfscp-r-keep-goingpersistent-500730694650/002/output.txt
        @@ -1,12 +1,8 @@
         
         >>> cp_r localdir dbfs:/Volumes/main/default/keep-going/dst --concurrency 2
        -Copy incomplete: 3 copied, 0 skipped (already existed), 1 failed, 0 not attempted.
        -Failed:
        -  localdir/f3.txt -> dbfs:/Volumes/main/default/keep-going/dst/f3.txt: HTTP 500 Internal Server Error
        -Error: 1 file was not copied
        +Error: databricks-api error: 
         localdir/f1.txt -> dbfs:/Volumes/main/default/keep-going/dst/f1.txt
         localdir/f2.txt -> dbfs:/Volumes/main/default/keep-going/dst/f2.txt
        -localdir/f4.txt -> dbfs:/Volumes/main/default/keep-going/dst/f4.txt
         
         Exit code: 1
         
        @@ -16,5 +10,4 @@
         PUT /api/2.0/fs/files/Volumes/main/default/keep-going/dst/f1.txt overwrite=false
         PUT /api/2.0/fs/files/Volumes/main/default/keep-going/dst/f2.txt overwrite=false
         PUT /api/2.0/fs/files/Volumes/main/default/keep-going/dst/f3.txt overwrite=false
        -PUT /api/2.0/fs/files/Volumes/main/default/keep-going/dst/f3.txt overwrite=false
         PUT /api/2.0/fs/files/Volumes/main/default/keep-going/dst/f4.txt overwrite=false
TestAccept/cmd/fs/cp-r-keep-going/single-500: branch ✅ | main (4e503ea) ❌ | latest (v1.26.0) ❌

main (4e503ea):

acceptance_test.go:1380: Diff:
        --- cmd/fs/cp-r-keep-going/single-500/output.txt
        +++ /tmp/TestAcceptcmdfscp-r-keep-goingsingle-500748650408/002/output.txt
        @@ -1,20 +1,7 @@
         
         >>> cp_r localdir dbfs:/Volumes/main/default/keep-going/dst --concurrency 2
        -Copied 6 files.
        +Error: databricks-api error: Fault injected by test.
         localdir/f1.txt -> dbfs:/Volumes/main/default/keep-going/dst/f1.txt
         localdir/f2.txt -> dbfs:/Volumes/main/default/keep-going/dst/f2.txt
        -localdir/f3.txt -> dbfs:/Volumes/main/default/keep-going/dst/f3.txt
        -localdir/f4.txt -> dbfs:/Volumes/main/default/keep-going/dst/f4.txt
        -localdir/f5.txt -> dbfs:/Volumes/main/default/keep-going/dst/f5.txt
        -localdir/f6.txt -> dbfs:/Volumes/main/default/keep-going/dst/f6.txt
         
        -=== Six uploads plus one retry
        -
        ->>> print_requests.py --summary --sort //api/2.0/fs/files/
        -PUT /api/2.0/fs/files/Volumes/main/default/keep-going/dst/f1.txt overwrite=false
        -PUT /api/2.0/fs/files/Volumes/main/default/keep-going/dst/f2.txt overwrite=false
        -PUT /api/2.0/fs/files/Volumes/main/default/keep-going/dst/f3.txt overwrite=false
        -PUT /api/2.0/fs/files/Volumes/main/default/keep-going/dst/f3.txt overwrite=false
        -PUT /api/2.0/fs/files/Volumes/main/default/keep-going/dst/f4.txt overwrite=false
        -PUT /api/2.0/fs/files/Volumes/main/default/keep-going/dst/f5.txt overwrite=false
        -PUT /api/2.0/fs/files/Volumes/main/default/keep-going/dst/f6.txt overwrite=false
        +Exit code: 1
        
FAIL

latest (v1.26.0): same output as main (4e503ea)

TestAccept/cmd/fs/cp-r-requests: branch ✅ | main (4e503ea) ❌ | latest (v1.26.0) ❌

main (4e503ea):

acceptance_test.go:1380: Diff:
        --- cmd/fs/cp-r-requests/output.txt
        +++ /tmp/TestAcceptcmdfscp-r-requests1431536269/002/output.txt
        @@ -1,6 +1,5 @@
         
         >>> [CLI] fs cp -r localdir dbfs:/Volumes/main/default/fs-cp-r-requests/dst
        -Copied 3 files.
         localdir/a.txt -> dbfs:/Volumes/main/default/fs-cp-r-requests/dst/a.txt
         localdir/b.txt -> dbfs:/Volumes/main/default/fs-cp-r-requests/dst/b.txt
         localdir/sub/c.txt -> dbfs:/Volumes/main/default/fs-cp-r-requests/dst/sub/c.txt
        @@ -8,6 +7,9 @@
         === Recursive copy: a directory PUT per directory, a file PUT per file, no directory HEAD per file
         
         >>> print_requests.py --sort //api/2.0/fs/
        +{"method":"HEAD","path":"/api/2.0/fs/directories/Volumes/main/default/fs-cp-r-requests/dst"}
        +{"method":"HEAD","path":"/api/2.0/fs/directories/Volumes/main/default/fs-cp-r-requests/dst"}
        +{"method":"HEAD","path":"/api/2.0/fs/directories/Volumes/main/default/fs-cp-r-requests/dst/sub"}
         {"method":"PUT","path":"/api/2.0/fs/directories/Volumes/main/default/fs-cp-r-requests/dst"}
         {"method":"PUT","path":"/api/2.0/fs/directories/Volumes/main/default/fs-cp-r-requests/dst/sub"}
         {"method":"PUT","path":"/api/2.0/fs/files/Volumes/main/default/fs-cp-r-requests/dst/a.txt"}

latest (v1.26.0): same output as main (4e503ea)

TestAccept/cmd/fs/cp/dir-to-dir: branch ✅ | main (4e503ea) ❌ | latest (v1.26.0) ❌

main (4e503ea):

acceptance_test.go:1380: 
        	Error Trace:	/home/runner/work/cli/cli/libs/testdiff/testdiff.go:22
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:1380
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:1121
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:666
        	Error:      	Not equal: 
        	            	expected: "\n>>> [CLI] fs cp -r localdir dbfs:/Volumes/main/default/fs-cp-test-[UNIQUE_NAME]/uploaded-dir\nCopied 2 files.\nlocaldir/file1.txt -> dbfs:/Volumes/main/default/fs-cp-test-[UNIQUE_NAME]/uploaded-dir/file1.txt\nlocaldir/file2.txt -> dbfs:/Volumes/main/default/fs-cp-test-[UNIQUE_NAME]/uploaded-dir/file2.txt\n\n>>> [CLI] fs cat dbfs:/Volumes/main/default/fs-cp-test-[UNIQUE_NAME]/uploaded-dir/file1.txt\nfile1 content\n\n>>> [CLI] fs cat dbfs:/Volumes/main/default/fs-cp-test-[UNIQUE_NAME]/uploaded-dir/file2.txt\nfile2 content\n"
        	            	actual  : "\n>>> [CLI] fs cp -r localdir dbfs:/Volumes/main/default/fs-cp-test-[UNIQUE_NAME]/uploaded-dir\nlocaldir/file1.txt -> dbfs:/Volumes/main/default/fs-cp-test-[UNIQUE_NAME]/uploaded-dir/file1.txt\nlocaldir/file2.txt -> dbfs:/Volumes/main/default/fs-cp-test-[UNIQUE_NAME]/uploaded-dir/file2.txt\n\n>>> [CLI] fs cat dbfs:/Volumes/main/default/fs-cp-test-[UNIQUE_NAME]/uploaded-dir/file1.txt\nfile1 content\n\n>>> [CLI] fs cat dbfs:/Volumes/main/default/fs-cp-test-[UNIQUE_NAME]/uploaded-dir/file2.txt\nfile2 content\n"
        	            	
        	            	Diff:
        	            	--- Expected
        	            	+++ Actual
        	            	@@ -2,3 +2,2 @@
        	            	 >>> [CLI] fs cp -r localdir dbfs:/Volumes/main/default/fs-cp-test-[UNIQUE_NAME]/uploaded-dir
        	            	-Copied 2 files.
        	            	 localdir/file1.txt -> dbfs:/Volumes/main/default/fs-cp-test-[UNIQUE_NAME]/uploaded-dir/file1.txt
        	Test:       	TestAccept/cmd/fs/cp/dir-to-dir
        	Messages:   	cmd/fs/cp/dir-to-dir/output.txt vs /tmp/TestAcceptcmdfscpdir-to-dir1707685595/002/output.txt

latest (v1.26.0): same output as main (4e503ea)

@eng-dev-ecosystem-bot

eng-dev-ecosystem-bot commented Nov 18, 2026 •

Copy link
Copy Markdown
Collaborator

✅ Integration test report

Commit: 5dbeb76

Run: 44387506379

Env 💚​RECOVERED ✅​pass 🙈​skip Time
💚​ aws linux-2core-8gb 1 271 18 11:07
💚​ aws-windows-latest-4core-16gb 1 273 16 4:48
💚​ azure linux-2core-8gb 1 270 18 9:27
💚​ azure-windows-latest-4core-16gb 1 272 16 4:57
💚​ gcp linux-2core-8gb 1 271 18 8:52
💚​ gcp-windows-latest-4core-16gb 1 273 16 4:39
Test Name aws linux-2core-8gb aws-windows-latest-4core-16gb azure linux-2core-8gb azure-windows-latest-4core-16gb gcp linux-2core-8gb gcp-windows-latest-4core-16gb
💚​ TestAccept 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R
Top 7 slowest tests (at least 2 minutes):
duration env testname
4:02 aws linux-2core-8gb TestAccept
3:56 azure linux-2core-8gb TestAccept
3:50 gcp linux-2core-8gb TestAccept
3:22 aws linux-2core-8gb TestFilerWorkspaceFilesExtensionsStat
3:02 aws linux-2core-8gb TestFilerRecursiveDelete/workspace_files_extensions
2:40 aws-windows-latest-4core-16gb TestFilerWorkspaceFilesExtensionsReadDir
2:18 azure linux-2core-8gb TestImportDirDoesNotOverwrite

During an outage every file fails, so keeping going would only work
through the rest of the copy one failure at a time. If files keep
failing and none has been copied for 5 minutes, the copy now starts no
new files, lets in-flight ones finish and reports the rest as not
attempted. Files that already failed once keep their error. A slow copy
with no failures never stalls, and a created directory counts as
progress.

The timeout is internal, not a flag. DATABRICKS_FS_CP_STALL_TIMEOUT_MS is
an undocumented test-only override. The clock is injectable, and unit
tests use a fake one.

Co-authored-by: Isaac <no-reply@databricks.com>
@yuanjieding-db
yuanjieding-db force-pushed the yuanjieding-db/fs-cp-stall-timeout branch from cd40cbd to 5dbeb76 Compare November 18, 2026 20:05
@yuanjieding-db yuanjieding-db changed the title fs cp -r: stop after 5 minutes without progress fs cp -r: stop when files keep failing for 5 minutes Nov 18, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants