Repository navigation
filer: describe Files API errors without a message, retry reads on 500 - #7839
Conversation
files/v2 errors print as "databricks-api error: " plus the message from the JSON body, so a failed HEAD (no body) or a non-JSON error page prints nothing else. Show the operation, path and HTTP status instead, and retry read-only calls (metadata, download, list) on 500 as well. Co-authored-by: Isaac <no-reply@databricks.com>
…code Address review: stop after 3 retries on 500 so a 500 that does not clear fails in seconds; describe errors from the multipart upload path too; include the body's error_code when there is one; no trailing space for non-standard status codes; changelog PR link. Co-authored-by: Isaac <no-reply@databricks.com>
✅ Integration test reportCommit: 3fd4cb7
Top 4 slowest tests (at least 2 minutes):
|
There was a problem hiding this comment.
Automated review at 3fd4cb7. Each finding was verified against the code.
Not ready yet: 1 blocking issue(s), tagged [blocking].
This PR improves Files API error messages (showing the HTTP status and path when the SDK error body is empty) and retries read-only calls on HTTP 500. The error-description and retry logic are sound, but newReadRetrier has a composite-literal bug that prevents compilation.
|
|
||
| func (w *FilesClient) newReadRetrier() ops.Retrier { | ||
| inner := apiretry.NewRetrier(w.retryBackoff, apiretry.RetrierConfig{StatusCodes: filesReadRetryStatusCodes}) | ||
| return &readRetrier{Retrier: inner} |
There was a problem hiding this comment.
[blocking] readRetrier embeds cappedRetrier, which in turn embeds ops.Retrier. Retrier is therefore a promoted field of readRetrier, and Go forbids promoted fields as keys in composite literals ("Promoted fields act as if they were ordinary fields of a struct except that they cannot be used as field names in composite literals of the struct"). This is a compile error. Compare line 232 where cappedRetrier{Retrier: ...} works because Retrier is a direct field there. Should be &readRetrier{cappedRetrier: cappedRetrier{Retrier: inner}}.
| var body struct { | ||
| ErrorCode string `json:"error_code"` | ||
| } | ||
| if json.Unmarshal(e.err.HTTPBody(), &body) == nil && body.ErrorCode != "" { |
There was a problem hiding this comment.
[nit] json.Unmarshal(e.err.HTTPBody(), &body) runs each time Error() is called (logging, fmt.Errorf wrapping, test assertions). Consider extracting the error_code once in describeFilesError and storing it in filesAPIError.
andrewnester
left a comment
There was a problem hiding this comment.
No blocking findings from a full pass; I'll take a manual look at the filer changes before approving.
Changes
FilesClient(UC Volumes):error_codeif it has one, e.g.stat /Volumes/c/s/v/dir: Files API returned HTTP 500 Internal Server Error. Before, it printed asdatabricks-api error:with nothing after it. This covers every FilesClient call, the experimental multipart upload included.Why
The files/v2 SDK prints an
APIErroras"databricks-api error: " + message, and the message comes only from a JSON error body. A failed HEAD (GetFileMetadata,GetDirectoryMetadata, used byStatand byWrite's parent-directory check) has no body, and a proxy error page is not JSON. Either way, users and CI seeError: databricks-api error:with no status, call or path. The old databricks-sdk-go usedhttp.StatusTextin that case. The filer moved to files/v2 in #6760 (2026-09-21), and the first empty-message failures on the dashboard date from that day.Since then the dashboard has 47 errors (one per test) with exactly this text: 179 failing runs in 60 days across UC Volumes tests in
integration/cmd/fsandintegration/libs/filer, on AWS, Azure and GCP. Some happened after #7559 and #7607, which already retry 408/429/502/503/504. In the run linked below the test failed in under 12 seconds, so the 10 retries were not exhausted. The status was outside the retry set, most likely 500.Decisions:
sdk-go/corev0.1.0 (the latest) still has a TODO inAPIError.Error().APIErroris replaced. A wrapped one already has its wrapper's context, and the replacement keepsUnwrap, so status-based mapping (httpStatus) still works.Tests
TestFilesClientRetriesReadOnlyCallsOn500: HEAD (directory and file) and GET return 500 with an empty body twice and then succeed. Fails without the fix.TestFilesClientDoesNotRetryDeleteOn500andTestFilesClientDescribesErrorWithoutMessagecheck the new text, a single DELETE attempt, and thathttpStatusstill sees the status. Without the fix both showdatabricks-api error:, the text from CI.TestFilesClientStopsRetrying500: a 500 that does not clear gets 3 retries.TestFilesAPIErrorText: anerror_code-only body and a non-standard status (520).TestDescribeFilesErrorLeavesOtherErrors: errors with a message, wrapped errors and nil are unchanged.go test ./libs/filer/ ./cmd/fs/and localTestAccept/cmd/fspass.integration/cmd/fsTestFsCpDir,TestFsStat*,TestFsLs,TestFsRmDirRecursivelyandTestFsCpFileToFileFileNotOverwrittenpass, all variants (26 subtests). A 500 cannot be forced on cloud.Dashboard
Error: databricks-api error:inTestFsCpDir/dbfs_to_uc-volumes(GCP, main, 2026-11-18 and 2026-11-19)TestFsRmDirRecursively/uc-volumesTestFsCpFileToDirWithOverwriteFlag/uc-volumes_to_dbfsTestFsCpDirToDirFileNotOverwritten/uc-volumes_to_uc-volumesTest: TestFsCpDir/dbfs_to_uc-volumes
AI agent (opus5.5, isaac_claude).
This pull request and its description were written by Isaac.