Skip to content

filer: describe Files API errors without a message, retry reads on 500 - #7839

Merged
denik merged 2 commits into
mainfrom
filer-files-api-empty-errors
Nov 20, 2026
Merged

denik merged 2 commits into
mainfrom
filer-files-api-empty-errors

Conversation

@denik

@denik denik commented Nov 20, 2026 •

Copy link
Copy Markdown
Contributor

Changes

FilesClient (UC Volumes):

  • When a Files API error carries no message, the error now names the operation, the path and the HTTP status, plus the body's error_code if it has one, e.g. stat /Volumes/c/s/v/dir: Files API returned HTTP 500 Internal Server Error. Before, it printed as databricks-api error: with nothing after it. This covers every FilesClient call, the experimental multipart upload included.
  • Read-only calls (file and directory metadata, download, list) are also retried on HTTP 500, at most 3 times per call. Writes, deletes and mkdir keep the current retry set (408/429/502/503/504).

Why

The files/v2 SDK prints an APIError as "databricks-api error: " + message, and the message comes only from a JSON error body. A failed HEAD (GetFileMetadata, GetDirectoryMetadata, used by Stat and by Write's parent-directory check) has no body, and a proxy error page is not JSON. Either way, users and CI see Error: databricks-api error: with no status, call or path. The old databricks-sdk-go used http.StatusText in that case. The filer moved to files/v2 in #6760 (2026-09-21), and the first empty-message failures on the dashboard date from that day.

Since then the dashboard has 47 errors (one per test) with exactly this text: 179 failing runs in 60 days across UC Volumes tests in integration/cmd/fs and integration/libs/filer, on AWS, Azure and GCP. Some happened after #7559 and #7607, which already retry 408/429/502/503/504. In the run linked below the test failed in under 12 seconds, so the 10 retries were not exhausted. The status was outside the retry set, most likely 500.

Decisions:

  • Retry 500 only on read-only calls, where a repeated attempt cannot apply anything twice. A retried PUT or DELETE after a 500 that the backend did apply would turn into "already exists" or "not found". Those keep the current retry set, and the new message shows their status if they still fail.
  • At most 3 retries on 500 (the other statuses keep the cap of 10), so a 500 that does not clear, e.g. on one broken path, still fails within seconds instead of about 75s.
  • The response body of a non-JSON error (a proxy HTML page) is not included; the status and path are enough to tell which call failed.
  • I could not confirm the status code: the CI output has only the empty text. If the next failure shows a status other than 500, the retry part should be revisited; the message part is needed either way.
  • Errors are described in the filer, not in the SDK. sdk-go/core v0.1.0 (the latest) still has a TODO in APIError.Error().
  • Only a bare APIError is replaced. A wrapped one already has its wrapper's context, and the replacement keeps Unwrap, so status-based mapping (httpStatus) still works.

Tests

  • TestFilesClientRetriesReadOnlyCallsOn500: HEAD (directory and file) and GET return 500 with an empty body twice and then succeed. Fails without the fix.
  • TestFilesClientDoesNotRetryDeleteOn500 and TestFilesClientDescribesErrorWithoutMessage check the new text, a single DELETE attempt, and that httpStatus still sees the status. Without the fix both show databricks-api error: , the text from CI.
  • TestFilesClientStopsRetrying500: a 500 that does not clear gets 3 retries.
  • TestFilesAPIErrorText: an error_code-only body and a non-standard status (520).
  • TestDescribeFilesErrorLeavesOtherErrors: errors with a message, wrapped errors and nil are unchanged.
  • go test ./libs/filer/ ./cmd/fs/ and local TestAccept/cmd/fs pass.
  • On AWS, integration/cmd/fs TestFsCpDir, TestFsStat*, TestFsLs, TestFsRmDirRecursively and TestFsCpFileToFileFileNotOverwritten pass, all variants (26 subtests). A 500 cannot be forced on cloud.

Dashboard

error verbatim
error/1148 Error: databricks-api error: in TestFsCpDir/dbfs_to_uc-volumes (GCP, main, 2026-11-18 and 2026-11-19)
error/1204 same text in TestFsRmDirRecursively/uc-volumes
error/1175 same text in TestFsCpFileToDirWithOverwriteFlag/uc-volumes_to_dbfs
error/1157 same text in TestFsCpDirToDirFileNotOverwritten/uc-volumes_to_uc-volumes

Test: TestFsCpDir/dbfs_to_uc-volumes

cp_test.go:139: stderr: Error: databricks-api error: 
    Error:      	Received unexpected error:
                	databricks-api error: 
    Test:       	TestFsCpDir/dbfs_to_uc-volumes

AI agent (opus5.5, isaac_claude).

This pull request and its description were written by Isaac.

files/v2 errors print as "databricks-api error: " plus the message from
the JSON body, so a failed HEAD (no body) or a non-JSON error page prints
nothing else. Show the operation, path and HTTP status instead, and retry
read-only calls (metadata, download, list) on 500 as well.

Co-authored-by: Isaac <no-reply@databricks.com>
@denik denik added the ai-auto Deprecated: use factory-generated instead. label Nov 20, 2026
…code

Address review: stop after 3 retries on 500 so a 500 that does not clear
fails in seconds; describe errors from the multipart upload path too;
include the body's error_code when there is one; no trailing space for
non-standard status codes; changelog PR link.

Co-authored-by: Isaac <no-reply@databricks.com>
@eng-dev-ecosystem-bot

Copy link
Copy Markdown
Collaborator

✅ Integration test report

Commit: 3fd4cb7

Run: 44705206160

Env 💚​RECOVERED ✅​pass 🙈​skip Time
💚​ aws linux-2core-8gb 1 272 19 4:04
💚​ aws-windows-latest-4core-16gb 1 274 17 4:06
💚​ azure linux-2core-8gb 1 271 19 3:43
💚​ azure-windows-latest-4core-16gb 1 273 17 4:41
💚​ gcp linux-2core-8gb 1 272 19 4:20
💚​ gcp-windows-latest-4core-16gb 1 274 17 4:53
Test Name aws linux-2core-8gb aws-windows-latest-4core-16gb azure linux-2core-8gb azure-windows-latest-4core-16gb gcp linux-2core-8gb gcp-windows-latest-4core-16gb
💚​ TestAccept 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R
Top 4 slowest tests (at least 2 minutes):
duration env testname
4:52 gcp-windows-latest-4core-16gb TestAccept
4:39 azure-windows-latest-4core-16gb TestAccept
4:04 aws-windows-latest-4core-16gb TestAccept
2:40 gcp linux-2core-8gb TestAccept

@denik
denik marked this pull request as ready for review November 20, 2026 09:15
@denik
denik requested review from a team as code owners November 20, 2026 09:15
@denik
denik requested review from Sankalp-Mittal, andrewnester and maksim-zuev-db and removed request for maksim-zuev-db November 20, 2026 09:15

@review-oss-bot review-oss-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated review at 3fd4cb7. Each finding was verified against the code.

Not ready yet: 1 blocking issue(s), tagged [blocking].

This PR improves Files API error messages (showing the HTTP status and path when the SDK error body is empty) and retries read-only calls on HTTP 500. The error-description and retry logic are sound, but newReadRetrier has a composite-literal bug that prevents compilation.


func (w *FilesClient) newReadRetrier() ops.Retrier {
inner := apiretry.NewRetrier(w.retryBackoff, apiretry.RetrierConfig{StatusCodes: filesReadRetryStatusCodes})
return &readRetrier{Retrier: inner}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[blocking] readRetrier embeds cappedRetrier, which in turn embeds ops.Retrier. Retrier is therefore a promoted field of readRetrier, and Go forbids promoted fields as keys in composite literals ("Promoted fields act as if they were ordinary fields of a struct except that they cannot be used as field names in composite literals of the struct"). This is a compile error. Compare line 232 where cappedRetrier{Retrier: ...} works because Retrier is a direct field there. Should be &readRetrier{cappedRetrier: cappedRetrier{Retrier: inner}}.

var body struct {
ErrorCode string `json:"error_code"`
}
if json.Unmarshal(e.err.HTTPBody(), &body) == nil && body.ErrorCode != "" {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[nit] json.Unmarshal(e.err.HTTPBody(), &body) runs each time Error() is called (logging, fmt.Errorf wrapping, test assertions). Consider extracting the error_code once in describeFilesError and storing it in filesAPIError.

@andrewnester andrewnester left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No blocking findings from a full pass; I'll take a manual look at the filer changes before approving.

@denik denik added factory-generated and removed ai-auto Deprecated: use factory-generated instead. labels Nov 20, 2026
@denik
denik added this pull request to the merge queue Nov 20, 2026
Merged via the queue into main with commit d92dc44 Nov 20, 2026
30 checks passed
@denik
denik deleted the filer-files-api-empty-errors branch November 20, 2026 14:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants