Repository navigation
fs cp -r: keep going past failed files - #7712
Draft
yuanjieding-db wants to merge 1 commit into
Draft
yuanjieding-db wants to merge 1 commit into
yuanjieding-db wants to merge 1 commit into
Conversation
yuanjieding-db
added a commit
that referenced
this pull request
Nov 17, 2026
Co-authored-by: Isaac <no-reply@databricks.com>
Contributor
|
Commit: Regression test reportSelected 4 changed tests (limit=20, 0 not selected) 4 tests: regression=4 unreleased=0 coverage=0 failing=0
TestAccept/cmd/fs/cp-r-keep-going/persistent-500: branch ✅ | main (4e503ea) ❌ | latest (v1.26.0) ❌main (4e503ea): latest (v1.26.0): TestAccept/cmd/fs/cp-r-keep-going/single-500: branch ✅ | main (4e503ea) ❌ | latest (v1.26.0) ❌main (4e503ea): latest (v1.26.0): TestAccept/cmd/fs/cp-r-requests: branch ✅ | main (4e503ea) ❌ | latest (v1.26.0) ❌main (4e503ea): latest (v1.26.0): same output as main (4e503ea) TestAccept/cmd/fs/cp/dir-to-dir: branch ✅ | main (4e503ea) ❌ | latest (v1.26.0) ❌main (4e503ea): latest (v1.26.0): same output as main (4e503ea) |
Collaborator
✅ Integration test reportCommit: 7d8cf4b
Top 5 slowest tests (at least 2 minutes):
|
yuanjieding-db
force-pushed
the
yuanjieding-db/fs-cp-keep-going
branch
from
November 18, 2026 18:12
da71748 to
4767c2b
Compare
yuanjieding-db
changed the base branch from
yuanjieding-db/fs-cp-skip-parent-check
to
yuanjieding-db/testserver-fault-headers
November 18, 2026 18:13
yuanjieding-db
force-pushed
the
yuanjieding-db/testserver-fault-headers
branch
from
November 18, 2026 19:02
99ea713 to
3dd7522
Compare
yuanjieding-db
force-pushed
the
yuanjieding-db/fs-cp-keep-going
branch
from
November 18, 2026 19:02
4767c2b to
17ad766
Compare
This was referenced Nov 18, 2026
fs cp -r used to stop at the first failed file: one transient 500 or 503 in a copy of many small files ended the whole copy. The errgroup is replaced by a copy that runs up to --concurrency files at a time and keeps going: - A retryable failure (408, 429, any 5xx including a non-JSON body, or a network error, after the filer's own retries) defers the file. After the walk, each deferred file gets one more full attempt. - 400, 404 and other errors fail only that file. - 401, 403, cancellation and a failure to print stop the copy. The walk then only counts the files left, reported as not attempted. - A retry without --overwrite that finds the target already there skips it only if the sizes match: a failed DBFS write can leave a truncated file. A different size or a failed Stat fails the file. - The run ends with a summary on stderr and exits 1 if any file was not copied. Co-authored-by: Isaac <no-reply@databricks.com>
yuanjieding-db
force-pushed
the
yuanjieding-db/fs-cp-keep-going
branch
from
November 18, 2026 19:37
17ad766 to
7d8cf4b
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stack. Each PR is based on the one before it, so its diff shows only its own change.
This is part 3.
Changes
fs cp -rused to stop at its first failed file. This PR replaces the errgroup with a copy that runs up to--concurrencyfiles at a time and keeps going:--overwritethat finds the target already there skips it only if the sizes match. A failed DBFS write can leave a truncated file, so a different size, or a failedStat, fails the file with a hint to rerun with--overwrite. The retry never overwrites: a proxy 5xx can come before the server's existence check, so overwriting could replace a file the user asked to keep. A partial file of the same size isn't detected. That's acceptable because a failed DBFS write leaves a truncated prefix, which is shorter unless the failure came after the last block.Copied N files.(plus any skipped), which JSON output leaves out. Otherwise the summary lists the counts, the reason the copy stopped (if it did), and the first 10 failures with their HTTP status. The command then exits 1. Per-file lines on stdout are unchanged.Why
A copy of many small files sends hundreds of thousands of requests, so a few transient errors are expected. One of them used to end the whole copy: a 10,000-file copy died after about 2,500 files on a single 503. The filer's per-call retries absorb short throttling. This change handles what gets past them, starting with a 500, which the filer doesn't retry.
Validation
These are staging runs on
kartik-test: 123,404 files × 150 KiB, 124 directories, default--concurrency 8. They used the full stack before it was split (da7174822), with the same behavior. Both builds carried the same validation-only patch that logs Files API requests.HEAD /fs/directories(per-file check, removed in #7711)The breaker, the stall timeout and the directory retries never triggered on staging. Unit and acceptance tests in #7785 to #7787 cover them.
Smaller end-to-end checks on the top-of-stack build, against a new throwaway Volume on
kartik-test(since deleted):--overwrite: all 200 files skipped,Copied 0 files, skipped 200 that already existed., exit 0.--overwrite: all 200 copied, exit 0.kartik-test. The workspace setsdisable_legacy_dbfs, so everydbfs:/tmpaccess is denied, and the released v1.26.0 gets the same error. CI integration tests cover DBFS-root sources.--concurrency 1with a 5s stall timeout: the upload succeeds both before and after the stall fix in fs cp -r: stop when files keep failing for 5 minutes #7785 (exit 0, 67s and 70s). Downloading that directory at the default--concurrency 8fails: see the known limits.Known limits
kartik-testthat happened after 304 of 2,000 files.fs cpinstalls no signal handler, and adding one would change the exit code CLI-wide, so it's left out of this stack. The "Stopped: the copy was cancelled." summary appears only when the caller cancels the command's context, which a Ctrl-C in a terminal doesn't do today. A rerun without--overwriteskips the files that were already copied.unexpected EOFafter 12 to 14 MB of each 200 MiB file). The released v1.26.0 fails the same way and leaves the same partial files. This stack retries each file once, finds the partial local file, and reports it with the size-check message, so the fix is a rerun with--overwrite, or at--concurrency 1, which succeeded with all checksums matching. Downloads stream straight into the target, with no temp file and no resume, which is out of scope here.DATABRICKS_EXPERIMENTAL_MULTIPART_UPLOAD, which is off by default.Tests
TestRecursiveCopy: one table that checks the exact stderr summary of each scenario:Mkdir.TestClassifyErrorandTestDescribeError.cmd/fs/cp-r-keep-going/single-500(one 500, retried) andpersistent-500(a non-JSON 500 on every upload of one file). The old fail-fast unit test incp_test.gois replaced by the table.Self-review
Isaac review: no findings.
/code-review(medium), 9 findings, all answered, none changed:--overwriteskips. Fixed (also raised by the orchestrator): such a file is now reported as failed ("cancelled: the copy stopped before this file finished"). A rerun without--overwritecan still skip its partial target, as before this change.Mkdirerror stops the copy. Fixed in fs cp -r: retry transient directory errors #7787.This pull request and its description were written by Isaac.