Repository navigation
acc: drop create-recovery lookups from cloud request recordings - #7838
Conversation
When a Lakebase create is answered with a transient error, the CLI reads the resource (404) before it repeats the create. The proxy dropped the failed POST but kept the lookup GET, so out.requests goldens still flaked on cloud. gcp: TestAccept/bundle/resources/postgres_roles TestAccept/bundle/resources/postgres_catalogs/basic aws: TestAccept/bundle/resources/postgres_roles/basic Co-authored-by: Isaac <no-reply@databricks.com>
|
Commit: Regression test reportSelected 1 changed tests (limit=20, 0 not selected) 19 tests: regression=0 unreleased=0 coverage=19 failing=0
|
✅ Integration test reportCommit: df963fe
10 interesting tests: 9 MISS, 1 RECOVERED
Top 3 slowest tests (at least 2 minutes):
|
Match lookups only on *_id query values, rewrite out.requests.txt once per repeat, and stop treating reads as lookups once the create is repeated or the resource is found, so a later identical create keeps its POST. gcp: TestAccept/bundle/resources/postgres_roles TestAccept/bundle/resources/postgres_catalogs/basic aws: TestAccept/bundle/resources/postgres_roles/basic Co-authored-by: Isaac <no-reply@databricks.com>
There was a problem hiding this comment.
Automated review at df963fe. Each finding was verified against the code.
Not ready yet: part of the automated review did not complete.
This PR extends the acceptance-test request recorder to also drop the recovery GET lookup (answered with 404) that the CLI makes between a failed POST create and its retry, complementing the earlier #7771 which only dropped the failed POST itself. The logic is narrowly scoped to exact resource paths derived from _id query parameters, and resets properly when the lookup finds the resource (no retry follows). The new test cases in selftest/record_proxy_retry cover the 404-lookup-then-retry path and the found-resource-then-redeploy path. The change looks correct and well contained; no issues survived verification.
❌ Integration test reportCommit: f4a7fe5
33 interesting tests: 14 RECOVERED, 13 KNOWN, 6 FAIL
Top 50 slowest tests (at least 2 minutes):
|
Changes
Follow-up to #7771. In cloud runs the acceptance proxy now also drops the create-recovery lookup when the CLI repeats a failed create.
When a Lakebase create (project, branch, role, catalog) is answered with a transient error, the CLI reads the resource before it repeats the create, to see whether the failed request created it anyway (
recoverPostgresCreate, #7747). #7771 dropped the failed POST once it was repeated, but the lookup GET (answered with 404) stayed inout.requests.txt, so the request goldens still flaked. #7771 listed this case as not covered.The recorder now remembers, for a POST answered with a transient error, the resource paths it names (the request path plus one of its query values, e.g.
POST /api/2.0/postgres/projects?project_id=pnames/api/2.0/postgres/projects/p). A GET of such a path answered with 404 is dropped together with the failed attempt when the CLI repeats the POST, and logged asDropped from out.requests.txt, a lookup before a repeated create: GET <path>.Judgment calls:
*_idquery parameter), so concurrent requests for other resources are not touched. A more general rule ("drop every 404 GET between a failed request and its repeat") would also catch unrelated parallel reads, so I kept it to the resource the create names.postgres_branches/update_protectedandpostgres_databases/update. It comes from a plaindatabricks postgres get-branch/get-databasecommand that fails, not from a bundle retry, so it is not a recording problem.Why
Dashboard:
Warn: deploying resources.postgres_projects.my_project: retrying after 500 Internal Server Error from POST /api/2.0/postgres/projects, then an extra"method": "GET", "path": "/api/2.0/postgres/projects/test-pg-proj-[UNIQUE_NAME]"before the POST inout.requests.jsonWarn: deploying resources.postgres_branches.main: retrying after 500 Internal Server Error from POST /api/2.0/postgres/projects/test-pg-proj-[UNIQUE_NAME]/branches, then an extra"method": "GET", "path": "/api/2.0/postgres/projects/test-pg-proj-[UNIQUE_NAME]/branches/main"inout.requests.create.jsonTests: postgres_catalogs/basic/DMS=, postgres_roles/basic/DMS=, postgres_roles/update/DMS=.
All of these show
prepare_server.go: Dropped from out.requests.txt, repeated after a transient error: POST ..., so #7771 handled the POST; only the lookup was left.Tests
selftest/record_proxy_retry(Proxy = true, same proxy as cloud runs) gets a new case: a 500 on the project create, which the CLI recovers from with a lookup (404) and a repeat. The recording has one POST and no GET of the project; without the change it has the GET.fault.py --after-handler). The lookup finds the project, then the script destroys and deploys again in the same recording. Both POSTs stay recorded; without the reset the second deploy dropped the first POST.selftest/andbundle/resources/postgres_*acceptance tests pass.postgres_rolesandpostgres_catalogs/basicon gcp andpostgres_roles/basicon aws (they only hit the path on a transient backend error, so this checks nothing else changed).This pull request and its description were written by Isaac.