Repository navigation
acc: leave requests answered with a transient error out of cloud recordings - #7771
Merged
Merged
Conversation
…rdings On a real workspace the CLI repeats a request answered with a 5xx, 408 or 429, and the proxy recorded both attempts, so out.requests goldens failed whenever the workspace had a hiccup. The proxy now removes the failed attempt from out.requests.txt once its response arrives. cloud: TestAccept/selftest/record_cloud TestAccept/bundle/resources/postgres_roles/update TestAccept/bundle/resources/postgres_databases/basic Co-authored-by: Isaac <no-reply@databricks.com>
Contributor
|
Commit: Regression test reportSelected 1 changed tests (limit=20, 0 not selected) 10 tests: regression=0 unreleased=1 coverage=9 failing=0
TestAccept/selftest/record_proxy_retry: branch ✅ | main (61f1fc5) ✅ | latest (v1.26.0) ❌latest (v1.26.0): |
Collaborator
✅ Integration test reportCommit: 04a240b
Top 50 slowest tests (at least 2 minutes):
|
A request answered with a transient error is now removed from out.requests.txt only once the same request is recorded again, so a request the CLI does not repeat (e.g. a create that succeeded despite a 500 and was found by the recovery lookup) stays in the recording. cloud: TestAccept/selftest/record_cloud TestAccept/bundle/resources/postgres_roles/update TestAccept/bundle/resources/postgres_databases/basic Co-authored-by: Isaac <no-reply@databricks.com>
denik
marked this pull request as ready for review
November 18, 2026 13:52
denik
requested review from
Sankalp-Mittal,
andrewnester and
renaudhartert-db
November 18, 2026 13:52
andrewnester
approved these changes
Nov 18, 2026
cloud: TestAccept/selftest/record_cloud TestAccept/bundle/resources/postgres_roles/update TestAccept/bundle/resources/postgres_databases/basic Co-authored-by: Isaac <no-reply@databricks.com>
Collaborator
❌ Integration test reportCommit: acb677a
1035 interesting tests: 1011 MISS, 9 FAIL, 9 RECOVERED, 6 PANIC
Top 50 slowest tests (at least 2 minutes):
|
idelafuente-oc
pushed a commit
to idelafuente-oc/cli
that referenced
this pull request
Nov 20, 2026
…bricks#7838) ## Changes Follow-up to [databricks#7771](databricks#7771). In cloud runs the acceptance proxy now also drops the create-recovery lookup when the CLI repeats a failed create. When a Lakebase create (project, branch, role, catalog) is answered with a transient error, the CLI reads the resource before it repeats the create, to see whether the failed request created it anyway (`recoverPostgresCreate`, [databricks#7747](databricks#7747)). databricks#7771 dropped the failed POST once it was repeated, but the lookup GET (answered with 404) stayed in `out.requests.txt`, so the request goldens still flaked. databricks#7771 listed this case as not covered. The recorder now remembers, for a POST answered with a transient error, the resource paths it names (the request path plus one of its query values, e.g. `POST /api/2.0/postgres/projects?project_id=p` names `/api/2.0/postgres/projects/p`). A GET of such a path answered with 404 is dropped together with the failed attempt when the CLI repeats the POST, and logged as `Dropped from out.requests.txt, a lookup before a repeated create: GET <path>`. Judgment calls: - Only 404 lookups are dropped, and only once the create is repeated. When the lookup finds the resource (the failed create did create it), no repeat follows, and the failed POST plus the lookup stay recorded, as before. That case records a different sequence (it also waits for the default branch), so mirroring the normal flow there would mean dropping requests that did happen. A found resource also ends the recovery: the failed attempt no longer counts as pending, so a later identical create (deploy, destroy, deploy in one script) keeps its POST. Before this change that later POST would have dropped the first one. - Lookups are matched by exact path (POST path + "/" + the value of an `*_id` query parameter), so concurrent requests for other resources are not touched. A more general rule ("drop every 404 GET between a failed request and its repeat") would also catch unrelated parallel reads, so I kept it to the resource the create names. - Not covered here: [error/3773](http://go/cli/dashboard/error/3773?main=1&pr=1&end=2026-11-20T08:00:00Z&last=3d) ("The request could not be processed. Please try again later ...") on `postgres_branches/update_protected` and `postgres_databases/update`. It comes from a plain `databricks postgres get-branch` / `get-database` command that fails, not from a bundle retry, so it is not a recording problem. ## Why Dashboard: | error | snippet | |---|---| | [error/3788](http://go/cli/dashboard/error/3788?main=1&pr=1&end=2026-11-20T08:00:00Z&last=3d) | `Warn: deploying resources.postgres_projects.my_project: retrying after 500 Internal Server Error from POST /api/2.0/postgres/projects`, then an extra `"method": "GET", "path": "/api/2.0/postgres/projects/test-pg-proj-[UNIQUE_NAME]"` before the POST in `out.requests.json` | | [error/3667](http://go/cli/dashboard/error/3667?main=1&pr=1&end=2026-11-20T08:00:00Z&last=3d) | `Warn: deploying resources.postgres_branches.main: retrying after 500 Internal Server Error from POST /api/2.0/postgres/projects/test-pg-proj-[UNIQUE_NAME]/branches`, then an extra `"method": "GET", "path": "/api/2.0/postgres/projects/test-pg-proj-[UNIQUE_NAME]/branches/main"` in `out.requests.create.json` | Tests: [postgres_catalogs/basic/DMS=](http://go/cli/dashboard/test/acceptance.TestAccept/bundle/resources/postgres_catalogs/basic/DMS=?main=1&pr=1&end=2026-11-20T08:00:00Z&last=3d), [postgres_roles/basic/DMS=](http://go/cli/dashboard/test/acceptance.TestAccept/bundle/resources/postgres_roles/basic/DMS=?main=1&pr=1&end=2026-11-20T08:00:00Z&last=3d), [postgres_roles/update/DMS=](http://go/cli/dashboard/test/acceptance.TestAccept/bundle/resources/postgres_roles/update/DMS=?main=1&pr=1&end=2026-11-20T08:00:00Z&last=3d). All of these show `prepare_server.go: Dropped from out.requests.txt, repeated after a transient error: POST ...`, so databricks#7771 handled the POST; only the lookup was left. ## Tests - `selftest/record_proxy_retry` (`Proxy = true`, same proxy as cloud runs) gets a new case: a 500 on the project create, which the CLI recovers from with a lookup (404) and a repeat. The recording has one POST and no GET of the project; without the change it has the GET. - A second new case: a 504 on the project create after the handler created it (`fault.py --after-handler`). The lookup finds the project, then the script destroys and deploys again in the same recording. Both POSTs stay recorded; without the reset the second deploy dropped the first POST. - Local `selftest/` and `bundle/resources/postgres_*` acceptance tests pass. - The commit asks CI to run `postgres_roles` and `postgres_catalogs/basic` on gcp and `postgres_roles/basic` on aws (they only hit the path on a transient backend error, so this checks nothing else changed). This pull request and its description were written by Isaac. --------- Co-authored-by: Isaac <no-reply@databricks.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Changes
In cloud runs the acceptance proxy now records a request only once when the CLI repeats it after a transient error (408, 429 or 5xx). Requests are still written when they arrive, so the order is unchanged. The proxy remembers each attempt answered with a transient error. When an identical request (same method, path, query and body) arrives later, it removes the failed attempt from
out.requests.txtbefore writing the repeat, and logsDropped from out.requests.txt, repeated after a transient error: <method> <path>. A failed request that is not repeated stays recorded.This applies only behind the proxy: cloud runs and local
Proxy = truetests. With the local testserver, a 5xx only happens when a test injects one withfault.py, and those goldens show the failed attempt on purpose (create_504,create_transient_500, ...). They are unchanged.This is the request-log counterpart of
SuppressLines.warn_retry, which already keeps theretrying after ...warnings out ofoutput.txt.Judgment calls:
Why
Lakebase and UC endpoints answer with a 500/504 now and then. The CLI retries and the deploy succeeds, but the recorded request log had the request twice, and the
out.requests*.jsongolden failed:Warn: retrying after 500 Internal Server Error from GET /api/2.0/postgres/projects/test-pg-proj-[UNIQUE_NAME]/branches/main, then an extra"method": "GET", "path": "/api/2.0/postgres/projects/test-pg-proj-[UNIQUE_NAME]/branches/main"inout.requests.update.jsonretrying after 500 Internal Server Error from GET /api/2.0/postgres/projects/test-pg-proj-[UNIQUE_NAME], then an extra GET of the project inout.requests.destroy.jsonretrying after 500 Internal Server Error from PATCH /api/2.0/postgres/[MY_ENDPOINT_ID], then an extra PATCH inout.requests.update.jsonretrying after 504 Gateway Timeout from PATCH /api/2.1/unity-catalog/models/..., then an extra PATCH of the modelTests: postgres_roles/update/DMS=true, postgres_branches/replace_existing/DMS=, postgres_endpoints/update_autoscaling/DMS=true, registered_models/comment_cleared/DMS=true.
Tests
selftest/record_proxy_retry(Proxy = true, so requests go through the same proxy as in cloud runs):bundle/,selftest/andcmd/acceptance tests pass, exceptbundle/templates/pydabs/check-formatting, which needsruffand fails on this machine without the change too.selftest/record_cloud,postgres_roles/updateandpostgres_databases/basicon every cloud.This pull request and its description were written by Isaac.